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FOREWORD 

This Handbook is published by the National Aeronautics and Space Administration (NASA) as a 
guidance document to provide engineering information; lessons learned; possible options to 
address technical issues; classification of similar items, materials, or processes; interpretative 
direction and techniques; and any other type of guidance information that may help the 
Government or its contractors in the design, construction, selection, management, support, or 
operation of systems, products, processes, or services. 

This Handbook is approved for use by NASA Headquarters and NASA Centers, including 
Component Facilities and Technical and Service Support Centers. 

This Handbook establishes general guidance to assist in complying with the requirements and 
recommendations of NASA-STD-7009, Standard for Models and Simulations, including 
technical information, application instructions, data, recommended practices, procedures, and 
methods used in support of NASA-STD-7009. NASA Standards, by definition and intent, are 
constrained in their content to include requirements as to what must be accomplished within the 
scope of their use. This Handbook includes suggestions as to methods by which to satisfy those 
requirements. As modeling and simulation span a wide range of technical disciplines, not all 
methods are similarly applied across all types of models and simulations (M&S). 

Requests for information, corrections, or additions to this Handbook should be submitted via 
“Feedback” in the NASA Standards and Technical Assistance Resource Tool at 
htt p://standards.nasa.gov 


Michael G. Ryschkewitsch Approval Date 

NASA Chief Engineer 
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NASA HANDBOOK FOR MODELS AND SIMULATIONS: 

AN IMPLEMENTATION GUIDE FOR NASA-STD-7009 

1. SCOPE 

1.1 Purpose 

The purpose of this Handbook is to provide technical information, clarification, examples, 
processes, and techniques to help institute good modeling and simulation practices in the 
National Aeronautics and Space Administration (NASA). As a companion guide to NASA- 
STD-7009, Standard for Models and Simulations, this Handbook provides a broader scope of 
information than may be included in a Standard and promotes good practices in the production, 
use, and consumption of NASA modeling and simulation products. NASA-STD-7009 specifies 
what a modeling and simulation activity shall or should do (in the requirements) but does not 
prescribe how the requirements are to be met, which varies with the specific engineering 
discipline, or who is responsible for complying with the requirements, which depends on the size 
and type of project. A guidance document, which is not constrained by the requirements of a 
Standard, is better suited to address these additional aspects and provide necessary clarification. 

This Handbook stems from the Space Shuttle Columbia Accident Investigation (2003), which 
called for Agency-wide improvements in the “development, documentation, and operation of 
models and simulations” 1 that subsequently elicited additional guidance from the NASA Office 
of the Chief Engineer to include “a standard method to assess the credibility of the models and 
simulations.” 2 General methods applicable across the broad spectrum of model and simulation 
(M&S) disciplines were sought to help guide the modeling and simulation processes within 
NASA and to provide for consistent reporting of M&S activities and analysis results. From this, 
the standardized process for the M&S activity was developed. 

The major contents of this Handbook are the implementation details of the general M&S 
requirements of NASA-STD-7009, including explanations, examples, and suggestions for 
improving the credibility assessment of an M&S-based analysis. 

1.2 Applicability 

This Handbook is applicable to a broad audience, ranging from the variety of M&S practitioners 
(developers, users, and analysts, for example) and consumers of M&S-based products and 
analyses to technical reviewers of M&S activities and analyses. 

NASA-STD-7009 and this Handbook are intended for use by M&S practitioners, technical 
reviewers, decision makers, and others in the organization implementing, reviewing, using, or 


'A Renewed Commitment to Excellence: An Assessment of the NASA Agency-Wide Applicability of the Columbia 
Accident Investigation Board Report. B2005-100968, January 30, 2004. Retrieved April 22, 2013. 
http://www.nasa.gov/pdf/55691main Diaz 020204.pdf . 

2 NASA Office of the Chief Engineer (September 1, 2006). Guidance in the Development of NASA-STD-7009. 
(Memo) 
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receiving the results from an M&S-based analysis. Further, as NASA-STD-7009 is primarily 
focused toward the results of an M&S-based analysis, which may be used by a variety of people, 
both internal and external to a given implementing organization, this Handbook may be used by 
anyone, as in the following examples: 

a. In receiving a presentation of an M&S-based analysis, a decision maker may use the 
Worksheet (section 4.2) as a guide to a more complete understanding of the analysis. 

b. In substantiating an M&S product or analysis, a peer review team may use the 
Worksheet and Handbook to structure the results of a technical review. 

c. In conducting an analysis with an existing M&S, a user/analyst may use the 
Worksheet and Handbook as a guide to covering basic M&S topics, which may be addressed 
during a future technical review or presentation for decision making. 

d. During the course of an M&S activity, an M&S development team may use the 
Handbook to ensure meeting the minimal expectations of a product used for critical analysis. 

Further discussion of the key roles of responsibility is included in section 5.1 of this Handbook. 

Anyone may use NASA-STD-7009 or this Handbook in the course of their modeling and 
simulation activities; however, this use should apply to M&S that meet established risk criteria 
determined by program/project management in collaboration with the NASA Technical 
Authority as outlined in Appendix A of NASA-STD-7009. The application of many different 
types of M&S is possible in the creation of an analytical tool. While the elucidation of those 
types may be instructive, it is also most likely to be incomplete; therefore, the types of possible 
M&S are not included here but are discussed briefly in section 4.1 of this Handbook. 

NASA-STD-7009 applies to any point in the program/project lifecycle to which an M&S-based 
analysis may be applied. However, the expectations on the quality of the M&S products and 
analysis credibility will vary (most likely, improve) as the program/project matures. For 
example, the results from an M&S-based analysis in predicting the behavior of a Real World 
System (RWS) will likely be less precise and less accurate in the conceptual phase of a project 
than after several years of operations. A listing of the NASA program/project management 
phases is given in section 5.1.1 of this Handbook. 

NASA-STD-7009 also applies to any size M&S activity if the criticality of the analysis, based on 
the influence of the M&S to the decision and the decision consequence, warrants its application. 

This Handbook is approved for use by NASA Headquarters and NASA Centers, including 
Component Facilities and Technical and Service Support Centers. This Handbook may also 
apply to the Jet Propulsion Laboratory (JPL) or to other contractors, grant recipients, or parties to 
agreements only to the extent specified or referenced in their contracts, grants, or agreements. 
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This Handbook, or portions thereof, may be referenced in contract, program, and other Agency 
documents for guidance. When this Handbook contains procedural or process requirements, they 
may be cited in contract, program, and other Agency documents for guidance. 

2. APPLICABLE DOCUMENTS 

2.1 General 

The documents listed in this section are applicable to the guidance in the Handbook. 

2.1.1 The latest issuances of cited documents may apply unless specific versions are 
designated. 

2.1.2 Non-use of specific versions as designated shall be approved by the responsible 
Technical Authority. 

The applicable documents are accessible via the NASA Standards and Technical Assistance 
Resource Tool at http://standards.nasa. gov or may be obtained directly from the Standards 
Developing Organizations or other document distributors. 

2.2 Government Documents 
Department of Defense (DoD) 

DoD Modeling and Simulation (M&S) Glossary. Retrieved April 
12, 2013. 

http://www.msco.mil/documents/_4_DoD%20M&S%20Glossary% 

20Combined.pdf 

Conceptual Model Development and Validation, RPG special topic, 
September 2006. Retrieved March 9, 2012. http://vva.msco.mil/ 

MIL-STD-3022 Standard Practice Documentation for Verification, Validation, and 

Accreditation (VV&A) for Models and Simulations 

Environmental Protection Agency (EPA) 

EPA/100/K-09/003 Guidance on the Development, Evaluation, and Application of 

Environmental Models 

NASA 


Aerospace Safety Advisory Panel Annual Report for 2008. 
Aerospace Safety Advisory Panel Annual Report for 2009. 
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JWST-PLAN-006 1 65 

Bolognese, J. (2009). The FEMCI Book. Retrieved October 25, 
2011. htto://femci. esfc.nasa.eov/femcibook.html. 

James Webb Space Telescope (JWST) System Modeling and 
Analysis and JWST Models Validation, Verification and 
Calibration Plan (SE-18), D42916 Rev. B 

JWST-REF-002290 

James Webb Space Telescope Math Models Guidelines Document 
(SE16), September 19, 2007, D36124 Rev. C 

NASA TM-2011- 
215987 

A Credibility Assessment Scoring (CAS) Process for Mission Risk 
Management 

NASA-STD-7009 

Standard for Models and Simulations 

NASA/SP-2007-6105 

NASA Systems Engineering Handbook 

NASA/SP-20 10-576 

NASA Risk-Informed Decision Making Handbook 

NASA/SP-20 10-580 

NASA System Safety Handbook, Volume 1, System Safety 
Framework and Concepts for Implementation 

NASA/SP-20 11-3422 

NASA Risk Management Handbook 

NPD 1000.0 

Governance and Strategic Management Handbook 

NPR 7120.5 

NASA Space Flight Program and Project Management 
Requirements 

NPR 7150.2 

NASA Software Engineering Requirements 

NPR 8000.4 

Agency Risk Management Procedural Requirements 

NPR 8715.3 

NASA General Safety Program Requirements 


Office of Management and Budget (OMB) 

OMB Circular A-l 19 Federal Participation in the Development and Use of Voluntary 

Consensus Standards and in Conformity Assessment Activities 

Sandia National Laboratories 


SAND2003-3769 

Oberkampf, W.; Trucano, T.; and Hirsch, C. (February 2003). 
“Verification, Validation, and Predictive Capability in 
Computational Engineering and Physics.” 
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2.3 Non-Government Documents 

American Society of Mechanical Engineers (ASME) 

ASME V&V 10 Guide for Verification and Validation in Computational Solid 

Mechanics 

ASME V&V 20 Standard for Verification and Validation in Computational Fluid 

Dynamics and Heat Transfer 

Institute of Electrical and Electronics Engineers (IEEE) 

1 597. 1 IEEE Standard for Validation of Computational Electromagnetics 

Computer Modeling and Simulations 


Modus Operandi, Inc. 


Hagan, T; Walker, J. (2009). Conceptual Data Model Evolution in 
Joint Strike Fighter Autonomic Logistics Information System of 
Systems Engineering. 

The Aerospace Corporation 

TOR-20 1 0(859 1 )- 1 7 Baxter, Michael J . (20 1 0). Guidance for Space Program Modeling 

and Simulation. 

The New England Journal of Medicine 

Haynes, A.B.; Weiser, T.G.; Berry, W.R.; Lipsitz, S.R.; Breizat, A- 
H. S.; Dellinger, E.P.; Herbosa, T.; Joseph, S.; Kibatala, P.L.; 
Lapitan, M.C.M; Merry, A.F.; Moorthy, K.; Reznick, R.K.; Taylor, 
B.; and Gawande, A.A. (January 29, 2009). “A Surgical Safety 
Checklist to Reduce Morbidity and Mortality in a Global 
Population.” The New England Journal of Medicine. 360; Vol. 5, 
pp. 491-499. 


Other 


Kelton, W.D.; Sadowski, R.P.; and Sturrock, D.T. (2004). 
Simulation with Arena, Third Edition. New York: McGraw-Hill. 

Oberkampf, W.L.; Deland, S.M.; Rutherford, B.M.; Diegert, K.V.; 
and Alvin, K.F. (March 2002). “Error and Uncertainty in Modeling 
and Simulation.” Reliability Engineering and System Safety, Vol. 
75, Issue 3, pp. 333-357. 
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2.4 Order of Precedence 

This Handbook provides guidance for promoting good practices in the production, use, and 
consumption of modeling and simulation products but does not supersede nor waive established 
Agency requirements/guidance found in other documentation. 
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3. ACRONYMS AND DEFINITIONS 


3.1 Acronyms and Abbreviations 


> 

< 

< 


% 

+ 

± 

® 

CT 

SM 

TM 


1- D 

2- D 

3- D 
AHS 
AIAA 
ANSI 
ASAP 
ASC 
ASCE 
ASME 
BC 
BSTA 
C 

CA 

CAS 

CM 

CMMI 

COTS 

CRM 

D 

DES 

DoD 

DOF 

EPA 

FEM 

FEMCI 

ft 

GOTS 

GSFC 

H 


greater than 
less than 

less than or equal to 

minus 

percent 

plus 

plus or minus 
registered trademark 
sigma: standard deviation 
service mark 
trademark 
one dimensional 
two dimensional 
three dimensional 

The American Helicopter Society International 

American Institute of Aeronautics and Astronautics 

American National Standards Institute 

Aerospace Safety Advisory Panel 

American Standards Committee 

American Society of Civil Engineers 

American Society of Mechanical Engineers 

boundary conditions 

Backplane Stability Test Article 

Compliant (in the context of the Recommendations 

Compliance Matrix in Appendix D) 

California 

credibility assessment scale 
configuration management 
capability maturity model integration 
commercial off the shelf 
continuous risk management 
depth 

discrete event simulation 
Department of Defense 
degree of freedom 
Environmental Protection Agency 
finite element model 

Finite Element Modeling Continuous Improvement 
foot (feet) 

government off the shelf 
Goddard Space Flight Center 
height 
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HI 

Hawaii 

Hz 

hertz 

I/O 

input/output 

IC 

initial conditions 

ICD 

Interface Control Document 

IDD 

Interface Definition Document 

IEC 

International Electrotechnical Commission 

IEEE 

Institute of Electrical and Electronics Engineers 

in 

inch(es) 

ISO 

International Organization for Standardization 

ISS 

International Space Station 

IV&V 

independent verification and validation 

JEOD 

Johnson Space Center Engineering Orbital Dynamics 

JPL 

Jet Propulsion Laboratory 

JWST 

James Webb Space Telescope 

kip 

kilopound(s) (1,000 pounds-force) (4,448.2216 Newtons) 

KSC 

Kennedy Space Center 

LISA 

Laser Interferometer Space Antenna 

M&S 

models and simulations (See usage note in section 4 of this 
Handbook.) 

MIL 

military 

m 

meter(s) 

mm 

millimeter(s) 

MOTS 

modified off the shelf 

MRCR 

Model Review Certification Record 

MSL 

Mars Science Laboratory 

N/A 

not applicable 

N 

Newton(s) 

NASA 

National Aeronautics and Space Administration 

NASTRAN 

NASA structural analysis system 

NC 

not compliant 

NCSL 

National Conference of Standards Laboratories 

NESC 

NASA Engineering and Safety Center 

NIST 

National Institute of Standards and Technology 

NM 

New Mexico 

NPD 

NASA Policy Directive 

NPR 

NASA Procedural Requirements 

OMB 

Office of Management and Budget 

OPR 

Office of Primary Responsibility 

OTS 

off the shelf 

PDE 

partial differential equation 

pdf 

probability density function 

PMSA 

Primary Mirror Segment Assembly 

RIDM 

risk informed decision making 

RISC 

Risk-Informed Safety Case 

RMP 

risk management plan 
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RMS 

RPG 

RWS 

s/w 

SINDA 

SME 

SP 

SPIE 

STD 

SW 

SWE 

TM 

TMG 

TRL 

TSS 

V 

V&V 

VIM 

VV&A 

W 


root mean square 
Recommended Practices Guide 
real world system 
software 

Systems Improved Numerical Differencing Analyzer 
subject matter expert 
special publication 

The International Society for Optical Engineering 

standard 

software 

software engineering 
technical memorandum 
thermal model generation 
technology readiness level 
thermal synthesizer system 
volume 

verification and validation 
International Vocabulary of Metrology 
verification, validation, and accreditation 
width 


3.2 Definitions 

Abstraction : The process of selecting the essential aspects of a reference system to be 
represented in a model or simulation, while ignoring those aspects that are not relevant to the 
purpose of the model or simulation 3 

Aleatory Uncertainty : The inherent variation in the physical system; it is stochastic, 
irreducible. 

Analysis : Any post-processing or interpretation of the individual values, arrays, files of 
data, or suites of executions resulting from a simulation. Analysis spans the whole extent of the 
M&S process from the study of the RWS and/or its referents, the gathering and reduction of data 
from the RWS or accepted referents for incorporation into a model, the development of 
simulation scenarios, and the study and reduction of data from use of the M&S into 
recommendations for the RWS. 

Architectural Diagram : A visual representation including the essential elements of any 
system and their interrelationships, along with the influences of external (environmental) or 
interfacing elements. 


3 Adapted from Fidelity ISG Glossary, Vol. 3.0, Retrieved April 22, 2013. 
http://www.sisostds.org/DigitalLibrarv.aspx7CommaiKHCore Download&EntrvId=3 1 267. 
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Assumption : Asserting information as a basis for reasoning about a system. In modeling 
and simulation, assumptions are taken to simplify or focus certain aspects of a model with 
respect to the RWS or presume distinct values for certain parameters in a model. Any modeling 
abstraction carries with it the assumption that it does not significantly affect the intended uses of 
the M&S. 

Calibration : The process of adjusting numerical or modeling parameters in the model to 
improve agreement with a referent. 

Caveat : To include “an explanation to prevent misinterpretation” and “a modifying or 
cautionary detail to be considered when evaluating, interpreting, or doing something.” 4 

Code Coverage : A method employed to measure how thoroughly software is tested; is 
commonly expressed on a percentage basis. 5 

Computational Model : The numerical representation of the mathematical model. 

Conceptual Model : The collection of abstractions, assumptions, and descriptions of 
physical processes representing the behavior of the reality of interest from which the 
mathematical model or validation experiments can be constructed. (NASA-STD-7009, adapted 
from ASME V&V 10, Guide for Verification and Validation in Computational Solid 
Mechanics). 

Configuration Management : A management discipline applied over the product's 
lifecycle to provide visibility into and to control changes to performance and to functional and 
physical characteristics. (NPR 7120.5, NASA Space Flight Program and Project Management 
Requirements) 

Credibility : “The quality to elicit belief or trust in M&S results.” (NASA-STD-7009) 

De facto Standard : An M&S that has achieved a dominant market share, which includes 
commercial off-the-shelf (COTS) applications, open-source code, and in-house code, within the 
relevant community of practice, but has not been formally established or required by wide 
consensus, code, or law. 

Design of Experiments (or Experimental Design) : A series of tests in which purposeful 
changes are made to the input variables of a system or process and the effects on response 
variables are measured. It is applicable to both physical processes and computer simulation 
models. 6 


4 htto://www.merriam- webster.com/dictionarv/caveat . Retrieved April 22, 2013. 

5 https://modelingguru.nasa.gov/docs/DOC-1828 . Retrieved April 22, 2013. 

6 This definition is largely a direct quote from A Brief Introduction to Design of Experiments , by Jacqueline K. 
Telford. Retrieved April 22. 2013. httD://www.ihuapl.edu/techdigest/TP/td2703/telford.pdf . 
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Deterministic : A term describing a system whose time evolution can be predicted exactly. 

Environment of the System tor Real World System) : The set of elements external to a 
system. The RWS and its environment may interact through the exchange of properties. Only the 
interactions relevant to an analysis should be included in the M&S. 

Epistemic Uncertainty : A lack of knowledge of the quantities or processes identified with 
the system; it can be subjective, is reducible, and may be identified with model uncertainty. 

Exploded Diagram : An illustration or diagram of a construction showing its parts 
separately but in positions that indicate their proper relationships to the whole. 

Face Validation (or Validation by Review) : The process of judgment by peers. Subject 
Matter Experts (SMEs), and other stakeholders to ascertain if an M&S is (or if M&S results are) 
consistent with perceived system behavior. The process should include an assessment of the 
model assumptions and specifications (and the conceptual model) and is often used in the early 
stages of M&S development. 

Favorable Use : A phrase deeming the application of an M&S as meeting relevant 
acceptance criteria by the program/project in collaboration with the Technical Authority 
(Requirement 4.1.3 (a) of NASA-STD-7009). 

Formal Peer Review : A review sanctioned by the program/project and conducted in 
accordance with rules explicitly established by the reviewed or reviewing organization. 

Formal Training : Instructor-led training of at least the depth of a semester-long university 
course at the advanced undergraduate or graduate level. 

Framework : A set of assumptions, concepts, values, and practices constituting a way of 
viewing reality. For M&S, this may be a computing environment that integrates multiple 
interacting components on a single computer or across a distributed network. 7 

Informal Peer Review : A technical assessment not conducted pursuant to a process 
established by the reviewed or reviewing organization. 

Input Pedigree : A record of the traceability of data from its source through all aspects of 
its transmission, storage, and processing to its final form used in an M&S. Any changes from the 
real-world source data may be of significance to its pedigree. Ideally, this record includes 
important quality characteristics of the data at every stage of the process. 

Interface Definition Document/Interface Control Document (IDD/ICD1 : For all practical 
purposes, synonymous terms. In the M&S context, these are formal documents defining 
interfaces between models, elements of a model, or simulations. The documents are typically 
used to specify input/output (I/O) variables, units, coordinate systems, initial and boundary 
conditions, and other parameters necessary to link one M&S to another. 


7 A modification from http://www.answers.eom/toDic/framework#ixzzlCL7UTZYb . Retrieved April 22, 2013. 
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Key Input Data : Input to the model with high relevance to the analysis. 

Kriging : An interpolation technique in which the surrounding measured values are 
weighted to derive a predicted value for an unmeasured location. Weights are based on the 
distance between the measured points, the prediction locations, and the overall spatial 
arrangement among the measured points. 8 

Limits of Operation : The boundary of the set of parameters for which an M&S result is 
acceptable based on the program-/project-required outcomes of verification, validation, and 
uncertainty quantification. (NASA-STD-7009) 

Mathematical Model : The mathematical equations, boundary values, initial conditions, 
and modeling data needed to describe the conceptual model (adapted from ASME V&V 10). 

Maximum Likelihood : A method of parameter estimation that produces the highest 
probability estimate from a particular data set, given the probability distribution model. 9 

Model : A description or representation of a system, entity, phenomena, or process. 10 A 
model may be constructed from multiple sub-models; the sub-models and the integrated sub- 
models are all considered models. Likewise, any data that go into a model are considered part of 
the model. A model of a model (commonly called a metamodel), e.g., a response surface 
constructed from the results of M&S, is considered a model. 

Numerical Errors : Errors traceable to various sources, including but not limited to 
floating point precision, inherent in all computer systems and leading to round off, underflow, 
and overflow; truncation of infinite series expansions; and approximations of exact solutions 
inherent in all numerical methods, e.g., approximation of derivatives and integrals by algebraic 
operations on sampled continuous functions. * 11 

Peer Review : A technical assessment conducted by one or more persons of equal 
technical standing to person(s) responsible for the work being reviewed. 

Probabilistic : Pertaining to stochastic (non-deterministic) events, the outcome of which is 
described by a probability. 12 

Real World System : The actual system the model is representing for the analysis; refers 
to the real system operating in its real environment. The term RWS is used to differentiate 
between the “system under analysis” and the “modeling system used for the analysis.” 

Synonyms used in this Handbook, for contextual reasons, are “real system” and “reality of 
interest.” (NASA-STD-7009) 


8 httD://suPDort.esri.com/en/knowledgebase/GISDictionarv/term/kriging. Retrieved April 22, 2013. 

9 Adapted from NIST: http://www.itl.nist.gov/div898/handbook/apr/section4/apr4 1 2.htm. Retrieved April 22, 2013. 

10 Adapted from Banks, J., ed. (1998). Handbook of Simulation. New York: John Wiley & Sons 

1 1 Yang, W.Y.; Cao, W.; Chung, T.-S.; and Morris, J. (2005). Applied Numerical Methods Using MA TLAB®. 
Hoboken: John Wiley & Sons, Inc. 

12 NASA/SP-2009-569, Bayesian Inference for NASA Probabilistic Risk and Reliability Analysis 
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Recommended Practices : Guidelines developed by professional societies, best practices 
documented for specific simulation codes, and NASA Handbooks and Guidebooks. 

Referent : Data, information, knowledge, or theory against which simulation results can 
be compared. (NASA-STD-7009; adapted from ASME V&V 10). Note: A referent may be the 
RWS to which the analysis is directed, or it could be a similar or analogous system, whereby the 
closeness of the referent to the RWS becomes pertinent. 

Regression Testing : Software testing that seeks to uncover errors after changes to the 
program, e.g., bug fixes or new functionality, have been made. The intent of regression testing is 
to assure a change did not introduce new errors. 13 

Risk : The combination of the probability that a program or project will experience an 
undesired event and the consequences, impact, or severity of the undesired event if it were to 
occur. The probability and consequences may have associated uncertainties. (NASA-STD-7009, 
adapted from NPR 7120.5). 

Scenario : The description or definition of the relevant system and environmental 
assumptions, conditions, and/or parameters used to drive the course of events during the run of a 
simulation model. The scenario may include but is not limited to the set of initial conditions, a 
set of assumptions, the values of relevant parameters (including system and environmental 
conditions, locations and quantities of objects, entities, or resources), or a sequence of actions, 
which may be specified in the model itself. Running the model with the given scenario is the 
simulation. 

Sensitivity Analysis : The study of how the variation in the output of a model can be 
apportioned to different sources of variation in the model input and parameters. The Results 
Robustness of an M&S-based analysis is obtained via sensitivity analysis (NASA-STD-7009, 
adapted from Saltelli, 2005). 

Simulation : The imitation of the characteristics of a system, entity, phenomena, or 
process using a computational model. 

Stochastic : Involving or containing a random variable or variables. Involving chance or 
probability. 14 


13 Myers, Glenford. (2004). The Art of Software Testing. Hoboken: John Wiley & Sons, Inc. 

14 http://dictionarv.reference.com/browse/stochastic . Retrieved April 22, 2013. 
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Uncertainty : (a) The estimated amount or percentage by which an observed or calculated 
value may differ from the true value; 15 (b) A broad and general term used to describe an 
imperfect state of knowledge or a variability resulting from a variety of factors, including but not 
limited to lack of knowledge, applicability of information, physical variation, randomness or 
stochastic behavior, indeterminacy, judgment, and approximation (NASA-STD-7009, adapted 
from NPR 8715.3, NASA General Safety Program Requirements); (c) Non-negative parameter 
characterizing the dispersion of the quantity values being attributed to a measurand. 16 

Uncertainty Quantification : The process of identifying all relevant sources of 
uncertainties; characterizing them in all models, experiments, and comparisons of M&S results 
and experiments; and quantifying uncertainties in all relevant inputs and outputs of the 
simulation or experiment. (NASA-STD-7009) 

Unit Problem : A problem that captures one or more fundamental characteristics relevant 
to the current application required for accuracy in the M&S. 

Unit Testing : Any type of software testing conducted on the smallest meaningful, testable 
fragments of code to ensure the code behaves exactly as intended under various conditions. For 
procedural programming languages, such code fragments are generally functions or 
subroutines. 17 

Validation : The process of determining the degree to which a model or a simulation is an 
accurate representation of the real world from the perspective of the intended uses of the M&S. 

Verification : The process of determining if a computational model accurately represents 
the underlying mathematical model and its solution from the perspective of the intended uses of 
M&S. 


15 The American Heritage Dictionary of the English Language, 4th ed. (2006). Boston: Houghton Mifflin Co. 

16 International Vocabulary of Metrology (VIM) 3 - Basic and General Concepts and Associated Terms. Bureau 
International des Poids et Mesures, Joint Committee for Guides in Metrology 

l7 http://www.saravanansubramanian.com/Saravanan/Articles On Software/Entries/201 0/1/19 Unit Testing 101 F 
or Non-Programmers.html . Retrieved April 22, 2013. 
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4. INTRODUCTION 

Note: The acronym M&S is used in a variety of ways in the literature: model and simulation, 
model or simulation, models and/or simulations, modeling and simulation, and modeling and/or 
simulating. The acronym is additionally confusing in that the term “model” can be used as both a 
noun and a verb. In the development of NASA-STD-7009, the decision was to focus on the 
product of models and simulations rather than on the process of modeling and simulating. This is 
explicitly stated in the Foreword to NASA-STD-7009 and is used throughout the Standard in that 
sense. Shifting meanings in this Handbook would be inconsistent and confusing; therefore, the 
use of M&S in this Handbook, as in the Standard, refers to models and simulations. There are 
times when the singular or plural form is intended, which can be inferred from the context. 

4.1 Background of NASA-STD-7009 and NASA-HDBK-7009 Development 

NASA-STD-7009 holds a unique place in the world of modeling and simulation in that it is 
generally applicable to all types of M&S and all phases of development, though it is primarily 
focused on the results of an M&S-based analysis. Most M&S standards and recommended 
practices have either focused on a single type of M&S, e.g., structures, fluids, or electrical 
controls, or on a particular phase of M&S development, e.g., verification or validation. 
Considering that program/project management is confronted with numerous types of analyses 
with which to make critical decisions, a common framework for understanding the results and 
assessing the analysis credibility is appropriate. However, this is complicated by the vast 
differences across engineering systems, which have resulted in the slow adoption of 
NASA-STD-7009. 

After formal approval in July 2008, NASA-STD-7009 was delegated to the individual program, 
project, or M&S practitioner to adopt. While existing programs/projects were not required to 
adopt it, new programs and projects are to adopt it, depending on their needs and M&S-based 
analysis criticality. Initially, the only guidance provided was in the form of rudimentary training 
materials on the NASA Standards website and the Office of Primary Responsibility (OPR) 
contact information. The first M&S document produced related to NASA-STD-7009 was a 
checklist by JPL to provide guidance specific to their M&S activities: Model Review 
Certification Record (MRCR) IX . Through part of 2010, the Constellation Program included 
NASA-STD-7009 as an applicable document and was developing a Recommended Practices 
Guide (RPG) based on the document (Recommended Practices Guide for Modeling and 
Simulation Credibility Assessment (Constellation Program), Aegis Report No. VJ-NASA-09- 
RP002, Draft 2009 19 ). 

In the first 2 years of its existence as a voluntary Standard, many questions were generated 
during implementation of NASA-STD-7009, both general questions and those for specific 
engineering disciplines attempting to meet its requirements. NASA’s Aerospace Safety Advisory 
Panel (ASAP) also maintained a continuing interest in the implementation of the Standard as 
indicated in its Annual Report tor 2008 and Annual Report for 2009, as its development was a 


18 For more information about the MRCR (an Excel file), contact W.J. Bertch at JPL. 

10 VJ-NASA-09-RP002 was in review in the Constellation Program but was never approved. 
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direct result of a safety-related accident. The interest and questions regarding the practical 
implementation of NASA-STD-7009 provided the impetus to develop this Handbook, which was 
sponsored by the NASA Engineering and Safety Center (NESC) in December 2009. 

Initial efforts in the development of this Handbook included the review of related NASA 
documentation (software requirements, product data and lifecycle management requirements, 
and NESC procedures); other related U.S. Government documentation, including OMB Circular 
A-l 19, Voluntary Consensus Standards, DoD and Department of Energy M&S verification and 
validation (V&V) and uncertainty quantification guidance; EPA/100/K-09/003, Guidance on the 
Development, Evaluation, and Application of Environmental Models; and external M&S 
standards and guides, including ASME V&V 20, Standard for Verification and Validation in 
Computational Fluid Dynamics and Heat Transfer; ASME V&V 10; IEEE 1597.1, IEEE 
Standard for Validation of Computational Electromagnetics Computer Modeling and 
Simulations, and the Mission Assurance Improvement Workshop, “Guidance for Space Program 
Modeling and Simulation,” (Baxter, 2010). 

The development of the Handbook was initiated with several pathfinder evaluations of on-going 
NASA M&S projects: the Orion Service Module Tank Slosh Model, the Orion Crew Module 
Water Landing Model, the Ares Thrust Oscillation Model, and the Mars Science Laboratory 
(MSL) Powered Descent Model. The general findings from these pathfinder studies are that it is 
good to: 

a. Have a structured process to follow. 

b. Use consistent terminology. 

c. Evaluate an M&S-based analysis more broadly, i.e., beyond V&V, to include all 
Credibility Assessment Scale (CAS) factors. 

d. Understand the RWS project requirements relevant to the M&S. 

e. Define accuracy requirements to validate critical analysis models appropriately. 

f. Understand how the validation of an M&S can be improved. 

g. Cross-link CAS Factors to NASA-STD-7009 requirements. 

h. Address M&S limits of operation. 

i. Provide guidance on coupled models. 

The questions related to implementation of the requirements of NASA-STD-7009 by M&S 
practitioners, the additional emphasis on risk by the ASAP, the details of various aspects of 
M&S provided by other Government and professional organizations, and the findings from 
NASA pathfinder projects provide the basis for the development of this Handbook. While 
implementation of NASA-STD-7009 is initially perceived as complex, this is only a reflection of 
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the complexity of the M&S discipline. Besides the sheer depth of calculation accomplished in 
many M&S, the variety of M&S types and methods add to its complexity. The following are 
examples of the varieties of M&S: 

• M&S primarily based on differential equations and/or difference equations. 

• A relative geometry model of various objects over time. 

• Regression models from empirical data. 

• Various system data relationship models. 

The uniqueness in implementing the various types of M&S is left to the discipline accomplishing 
the M&S-based analysis, e.g., finite element analysis, system process analysis, or computational 
fluid dynamics, and to the relevant professional organizations, e.g., American Institute of 
Aeronautics and Astronautics (AIAA), ASME, or IEEE. This is not a full elucidation of the 
M&S disciplines that exist and is complicated by M&S systems that are combined into larger 
and/or distributed analytical platforms. Therefore, one essential consideration in the development 
of the Handbook was to provide guidance and explanations about the requirements and 
recommendations included in NASA-STD-7009 and thus ease and broaden its use. A worksheet- 
centered approach provides a readily usable tool for an M&S effort to use throughout its 
lifecycle. 

Worksheets and checklists are used in a variety of venues to ensure operations and processes are 
accomplished in an organized, consistent, and complete manner, which can improve both the 
safety and quality of the process. “NASA research has led to standardized checklist 
characteristics in the field of general aviation.” 20 Studies were also accomplished in the area of 
medical/surgical procedures showing the implementation of checklists had associated 
“reductions in the rates of death and complications among patients” (Haynes, et al., 2009). As 
NASA’s use of M&S can have safety and/or critical implications to human life or mission 
success, the use of a checklist or worksheet to guide the development, use, and discussion of 
M&S-based results is appropriate. The Worksheet resulting from the development of this 
Handbook combines aspects of both worksheets and checklists. 

Note: This Handbook and Worksheet are not intended to be comprehensive or overly 
prescriptive. It is not possible to include everything needed for every type of M&S or leave 
enough room for full disclosure of everything requested. This Handbook primarily provides 
guidance to a more complete discussion of the details surrounding M&S-based analyses. 

4.2 The Inferred NASA-STD-7009 Process 

A process for using NASA-STD-7009 was not defined because of the variety of possible 
implementations; however, a generalized process is inferred (figure 1, NASA-STD-7009 Inferred 
Process). 


20 httD://hwebbir.tvpeDad.com/openloops/2005/09/how to create a.html . Accessed April 23, 2013. 
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Figure 1 — NASA-STD-7009 Inferred Process 


This process begins with potential exclusion gates based on the M&S being embedded in control 
or flight software and whether the use is intended as a significant part of critical decisions. This 
comes with the understanding that any project developing or using an M&S may wish to make 
use of the defined practices. Once use of NASA-STD-7009 is established, responsibilities are set, 
and expectations for achieving the requirements of the M&S project, including what and how to 
manage the artifacts of the project, are substantiated. During the development and use of the 
M&S, control of the M&S, data, and analysis products is accomplished. From the M&S-based 
analyses, results are reported that lead to decisions made with respect to the RWS or the M&S. 

4.3 A Worksheet-Centered Approach 

A key tool facilitating the use of the NASA-STD-7009 and this Handbook is a Worksheet 
developed as a guide to ensure the operations and processes for M&S are accomplished in an 
organized, consistent, and complete manner, thus improving both the safety and quality of the 
process. 

4.3.1 Purpose of the Worksheet 

This Handbook is designed to help the users of NASA-STD-7009 focus on concepts generally 
applicable to the discipline of modeling and simulation and, to that end, uses a Worksheet- 
centered approach. For example, M&S should be verified and validated distinctly, i.e., 
separately, and the results of an M&S-based analysis should include an understanding of the 
associated uncertainties. The other items in the Worksheet, which includes a results credibility 
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assessment, can also be addressed by most M&S. However, some types or applications of M&S 
may have additional criteria for acceptance, which is the subject for discipline-specific 
recommended practices. Note: 

a. The use of a worksheet does not guarantee an error-free system or process, especially 
if “checking off the boxes” is the only result. The Worksheet provides a structure for reaching a 
more complete understanding of an M&S-based analysis, i.e., one that is well developed, 
documented, maintained, and operated. From the information obtained in progressing through 
the Worksheet, the M&S practitioner, technical reviewer, and/or decision maker will have a 
better sense of the M&S and analysis credibility. 

b. This Worksheet is not intended to capture all information relevant to a given 
M&S-based analysis. It is intended as a guide to a more complete understanding of the M&S and 
analysis results and allows the documentation of notes and other reference documentation. 

c. The Worksheet solicits the results from the assessment of the M&S activity’s 
compliance with the requirements of NASA-STD-7009 (in accordance with Appendix C of the 
Standard). 

4.3.2 Organization of the Worksheet 

The Worksheet is organized into a header and four sections, described in detail in section 5 of 
this Handbook. 

a. The header contains basic information about the RWS, the M&S, the assessed 
analysis criticality, and the responsible parties for the RWS and M&S. 

b. Each section includes key items and questions to elicit further details about the M&S 
and associated analysis and report summary information from those inquiries. 

(1) The first section focuses on the RWS to which an analysis is directed, the M&S 
that provides the basis for the analysis, and how the M&S and RWS correlate. 

(2) With this foundation, the second section documents the analysis results, the 
uncertainty contained in the results, and caveats that may detract from the 
acceptability of the analysis. 

(3) The third section emphasizes the credibility assessment as a minimal set of factors 
providing a broad assessment of the M&S results, including the aspects of 
development, management, and use. 

(4) The fourth section addresses compliance with the requirements of 
NASA-STD-7009 and the risk associated with accepting the analysis 
recommendations. 
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A key feature of this Worksheet is that it starts with an understanding of the context and 
criticality for a given situation and ends with an understanding of the risks involved with 
accepting the analysis recommendations derived. 

4.4 Standardized Items for Reporting an M&S-Based Analysis 

The following set of standardized items for reporting an M&S-based analysis was developed in 
NASA-STD-7009 and is explained in more detail in this Handbook: 

a. Understand the roles and responsibilities of the people involved, from the 
customers/stakeholders, i.e., the program/project managers and Technical Authorities, to the 
M&S practitioners and analysts. This responsibility chain follows current NASA governance. 

b. Plan and document the activities in the M&S process. A majority of the requirements 
of NASA-STD-7009 require documentation of what was accomplished. 

c. Perform a risk assessment in accordance with Appendix A of NASA-STD-7009 to 
determine the criticality of an impending analysis and the required use of the NASA-STD-7009. 

d. Understand the RWS and specific problem, the M&S, and how well they correlate. 

e. Report the following items with M&S-based analysis results. If these items or 
acceptability criteria for these items are not included, available, or accomplished, a clear 
statement of that situation is to be declared. 

(1) The estimated results. 

(2) A statement of uncertainty in the results. 

(3) Caveats to the results. Placards of non-nominal conditions or events in an 
M&S-based analysis should accompany the results if: 

A. Established acceptance criteria are not achieved. 

B. Model assumptions are violated. 

C. Limits of model operation are violated. 

D. Warning and/or error messages occur during the execution of an M&S. 

E. Intended use and setup/execution assessments are unfavorable. 

F. Requirements of NASA-STD-7009 are waived. 

(4) An understanding of results credibility. NASA-STD-7009 defines eight factors of 
credibility that contribute to the overall technical assessment of an M&S-based 
analysis. Depending on the criticality of an analysis, a technical review of an 
M&S activity may be prudent, using the following factors as a basis: 

A. Verification. 

B. Validation. 
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C. Input pedigree. 

D. Results uncertainty. 

E. Results robustness. 

F. Use history. 

G. M&S management. 

H. People qualifications. 

f. Understand the level of compliance with the requirements of NASA-STD-7009. 

Each M&S activity and program/project is unique, with a variety of expectations with respect to 
budget, schedule, requirements, and risk. These are to be balanced with the completeness with 
which an M&S activity complies with the requirements of NASA-STD-7009. 

g. Assess the risk associated with basing a decision on the analysis. Methods of 
assessing risk depend on the set of scenarios in which the risk is manifest, the likelihood of those 
scenarios, and the consequences if they occur (NASA/SP-201 0-576, NASA Risk-Informed 
Decision Making Handbook). Many factors contribute to how the M&S-based analysis 
influences the risk of a decision, e.g., the credibility factors listed in section 4.3.e.(4) above. 

NASA-STD-7009 and this Handbook provide a basis for good practices in modeling and 
simulation. They are not expected to address the details of all types of M&S or all problem 
domains, but they do provide a standardized structure and foundation from which further 
understanding is possible. 

4.5 Models - Key Concept 

NASA-STD-7009 defines a model as a description or representation of a system, entity, 
phenomena, or process. Models are necessarily imperfect and/or incomplete for a variety of 
reasons: 

a. It is not possible to make an exact representation because: 

( 1 ) The model would exceed the limits of the computational platform. 

(2) Details are not sufficiently characterized so as to be included in the model. 

(3) It is not possible to include all possible variations of the subject RWS. 

b. It is not desirable to make an exact representation because: 

(1) Added fidelity (detail) adds cost and complexity. 

(2) Adding unnecessary details detracts from focus of the analysis. 

As such, models are abstract representations of existing, proposed, or imagined systems; 
however, the intent is to include the pertinent representations necessary for the model’s intended 
purpose. The key concept is that models and simulations do not produce exact or perfectly 
correct results. Both the limitations and imperfections built into the model, i.e., epistemic 
uncertainty, and the inherent system variability included in the analysis, i.e., aleatory uncertainty, 
are manifested as uncertainty in the M&S results and need to be clearly understood. 
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In addition, there are also references to terms such as sub-model, linked model, coupled model, 
integrated model, surrogate model, and metamodel. NASA-STD-7009 discusses some of these 
terms, stating, “A model may be constructed from multiple sub-models; the sub-models and the 
integrated sub-models are all considered models. Likewise, any data that goes into a model is 
considered part of the model. A model of a model (commonly called a metamodel), e.g., a 
response surface constructed from the results of M&S, is considered a model.” 

Surrogate models are synonymous with metamodels in some instances, although there are other 
uses of the latter term that include the integration of sub-models and the linkage of stand-alone 
models, as described below. In effect, surrogate models are constructed in a manner parallel to 
empirical models where experimental data are used as the basis for I/O relationships between 
independent and dependent variables. In the case of surrogate models, the data come from the 
results of simulations rather than experiments. 

Sub-models and coupled models refer to elements of complex, aggregated models with two-way 
interaction between the elements that mirror the interactions between corresponding parts of the 
RWS. A typical example is a space vehicle Guidance, Navigation, & Control model, where sub- 
models representing the control system, sensors, actuators, vehicle dynamics, and 
intemal/extemal environments may interact through complex, multipath feedback loops. 

The term linked models refers to cases in which two or more models interact through one-way 
coupling. A typical example is found in the case of telescopes and optical instruments, where the 
impact of temperature changes upon optical image quality must be limited by careful design. The 
linked analysis required in this case involves (1) executing a simulation using a thermal model of 
the system, (2) transferring (mapping) the predicted temperatures to a structural model of the 
system, (3) executing a simulation of the temperature-induced elastic deformations of the 
structure using this structural model, (4) transferring the structural deformations into an optical 
model, (5) transferring the predicted temperatures to the optical model to account for 
temperature-dependent index-of-refraction of lens elements, if any, and (6) executing a 
simulation of the geometric and physical diffraction phenomena using the optical model. There 
are also cases in which individual models are developed and possibly used on their own and then 
integrated into a larger analytical model to address more system-wide issues. In either case, the 
recommendation is to apply NASA-STD-7009 to the individual M&S and also to the linked or 
integrated M&S as a whole. The level definitions for the Input Pedigree factor in the CAS 
anticipate exactly this scenario. 

4.5.1 NASA’s Motivation to Model 

In the development of aerospace systems outside NASA, e.g., in commercial aviation, the risk 
associated with models can be mitigated by hours of flight test in the operational environment. 
The nature of NASA’s missions often involves one-of-a-kind systems that have a high impact if 
unsuccessful, such as: 

a. Loss of human life. 

b. Loss of high-value equipment. 
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c. Loss of mission products, e.g., unique science. 

d. Limited reflight opportunities. 

e. Re-design of the system. 

Because of these impacts and a relatively high-risk profile, testing of operational systems in 
operational environments, e.g., flight tests, is typically limited. NASA’s engineering processes, 
therefore, depend on models of the system to a higher degree than is typically found in other 
industries to help mitigate operational risk. Thus, a methodical approach to accepting the results 
of these models is beneficial. 

4.5.2 The Modeling and Simulation Process 

No discussion of modeling and simulation is complete without an understanding of the process 
involved from conception through application. Many such processes for an M&S activity are 
published with nuances to the particular discipline to which each is directed. For NASA M&S 
activities, the process followed would be part of the M&S Plan identified in Requirement 4. 1 .4 
of NASA-STD-7009. A generalized process may be inferred from the original forays into M&S 
V&V. Some additional ideas inherent in this topic, along with the basic ideas of the RWS, 
conceptual model, and computational model, are that: 

a. A referent may be used as an analog to the RWS that does not currently exist and may 
take a variety of forms, e.g., information from a similar system or a system specification. In this 
case, validation of the computational/simulation model is accomplished with respect to the 
referent, and the correlation of the referent to the RWS needs to be evaluated. 

b. A model specification may be necessary along with a conceptual model, which may 
include the precise formalisms of mathematical notation or pseudo-code for complex constructs, 
i.e., a math model. 

c. A simulation uses a computational model or simulation model to predict, mimic, 
emulate, or represent system behavior. 

d. M&S-based analysis results are often derived, i.e., post-processed, from raw M&S 
output data. Synonymous terms to an analysis based on the use of a model or simulation are 
model-based analysis, M&S-based analysis, and simulation analysis. 

A generalized process for M&S development and use is shown in figure 2, Generalized M&S 
Process Including Model Use, in a left-to-right process flow. The process flow is described in 
table 1, Generalized M&S Processes, with the incumbent V&V feedback processes (shown in 
green). 
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Figure 2 — Generalized M&S Process Including Model Use 


Table 1 — Generalized M&S Processes 


M&S Process Step 

Description 

Importance in M&S 

1 . RWS (and/or Analogous 
Referent) 

The object/subject of the 
modeling/analysis 

The RWS is the focus of the 
M&S, from which the model is 
developed and to which the 
analyses are directed. 

2. Examine the System 

Decomposition analysis of the 
subject to determine the relevant 
aspects to include in the model 

Determines the abstractions, e.g., 
what to include and what to not 
include in the model, level of 
detail, assumptions, and model 
component interrelationships 

3. Conceptual Model and 
Model Specification 

The collection of abstractions, 
assumptions, and descriptions of 
physical processes representing 
the behavior of the reality of 
interest from which the 
mathematical model or validation 
experiments can be constructed 
(adapted from ASME V&V 10) 

Provides narrative descriptions, 
block diagram, flow chart, 
equations, or pseudo-code, 
which, in turn, provide a static 
representation of the RWS, 
which is most likely 
complemented by a model 
specification 

4. Validation of Conceptual 
Model 

The process of determining the 
degree to which the conceptual 
model is an accurate 
representation of the real world 
from the perspective of the 
intended uses of the model or the 
simulation 

Ensures the essential aspects of 
the RWS are captured for the 
impending analysis 

5. Model Implementation 

Constructing the models (for 
computer-based models, this 
means coding the algorithms) 
that will represent system 
behavior 

The RWS model is built from the 
specifications. 

6. Computational/Simulation 
Model 

The computer-based code and 
logic, including the numerical 
representation of the 
mathematical model that imitates 
the characteristics of a system, 
entity, phenomena, or process 

The Computational/Simulation 
Model is the basis for RWS 
analysis. 
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M&S Process Step 

Description 

Importance in M&S 

7. Verification of 

Computational/Simulation 
Model with Conceptual 
Model 

The process of determining if a 
computational model accurately 
represents the underlying 
conceptual and mathematical 
model and its solution from the 
perspective of the intended uses 
of M&S 

Ensures the model is built to 
specification 

8. Validation of 

Computational/Simulation 
Model with RWS or 
Analogous Referent 

The process of determining the 
degree to which a 
Computational/Simulation Model 
is an accurate representation of 
the real world from the 
perspective of the intended uses 
of the M&S 

Ensures the model adequately 
represents the intended RWS 

9. Develop and Run Scenarios 
(Design of Experiments) 

The description or definition of 
the relevant system and 
environmental assumptions, 
conditions, and/or parameters, 
i.e., M&S input, used to drive the 
course of events during the 
simulation 

The set of scenarios to run 
determines the completeness of 
the analysis and can support the 
sensitivity analysis of scenario 
parameters. 

10. M&S Output Data 

The data produced by the M&S 
when running the scenarios 

The raw output from running the 
M&S with the scenarios of 
interest; these data may be post- 
processed to produce results 
directly related to the analysis. 

1 1 . Results Analysis Applied to 
RWS 

The M&S output data analysis 
producing direct 
recommendations applicable to 
the RWS 

The M&S process comes full 
circle with implications to the 
RWS. 


4.5.3 Philosophy and Structure of this Handbook with a Worksheet 

NASA-STD-7009 is structured toward the management and quality aspects of M&S activities. 
Since the focus of this Handbook is toward more effective use of NASA-STD-7009, its structure 
is necessarily different and follows the structure of the Worksheet 21 . A diagrammatic overview is 
shown in figure 3, Overview of the Worksheet, with a full page view in Appendix A of this 
Handbook. Each portion of the Worksheet is discussed generally and then in detail. 


21 A spreadsheet version of the M&S Assessment Worksheet can be downloaded from 
https://standards.nasa.gov/released/nasa/NASA-HDBK-7009 Worksheet.xlsx. Retrieved July 9, 2013. 
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(See Appendix A of this Handbook for a full-page view of the Worksheet.) 

The top of the Worksheet includes a representational credibility assessment spider plot that is 
one of the key features of NASA-STD-7009. As the credibility of results is central to a more 
complete understanding of an M&S-based analysis, each credibility factor and its relative rating, 
e.g., on a spider plot or bar chart, are included. 

4.5,4 General Structure of the Sections Supporting the Worksheet 

The goal for these guidelines and for the use of the Worksheet is that they are ultimately 
useable/actionable, i.e., users know what they need to do. As such, instructions in section 5 of 
this Handbook follow the following outline: 

a. An introductory paragraph defining the Worksheet item. 

b. Succinct statements about what is needed to satisfy this item. People versed in 
NASA-STD-7009 and this Handbook would use these statements as a review of details to 
consider, while people new to M&S might use this as an overview with more detailed 
explanations and examples supporting these details in the following sections. 

c. Explanations as to the type of information to enter in the Result and Comments 
columns for this item/question and what additional information should be considered. 
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d. Examples of what to include for this Worksheet item. 

e. If the Worksheet item is a CAS factor, suggestions for achieving each defined level of 
credibility. 

In the initial development phases of this Handbook, the Worksheet was simply a list of 
questions. While benchmarking against checklists and worksheets found in other industries, the 
ideas of grouping related items and providing an approximate flow emerged. Once this was 
attempted, the structure of the resulting Worksheet developed. 
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5. THE WORKSHEET 

The information and questions included in this Worksheet are meant to induce a spirit of general 
M&S inquiry, which is by no means all inclusive or necessarily mandatory in all cases. These are 
recommended practices and suggested questions by which to glean more depth of understanding 
of the M&S-based analysis, while potentially working toward NASA-STD-7009 compliance. 


5.1 Header 


The Worksheet header (figure 4, Worksheet Header) addresses two areas of clarification for 
using any M&S-based analysis: linking the M&S used in the analysis to the RWS or sub-system 
to which the analysis is made and documenting the responsible parties for the RWS and the 
M&S. The clarification of the responsibility chain for the RWS and the M&S became apparent 
during some initial uses of NASA-STD-7009. 



NASA-STD-7009 
M&S Assessment Worksheet 



System 


Sub-System, Elemtut, or Aspect of System Under Analysis: 


System Lifecycle Phase: 


Respomldhy Cham: P/P Mgt & Tech Authority 


M&S: 

M&S Responsible Party: 


State of M&S: 

In Dcvcfapmeit / Operational 


Subject of Analysis (c.g.. Production, Ground Opt, Ffigfr, Mission, Entry, Descent, Landing): 


Risk Assessment Result (per NASA-STD-7009 Appends A)? 

7009 Use b Required / 7009 Use Is NOT Required / Not Perform*! 


Figure 4 — Worksheet Header 


5.1.1 Left Side of Header 

The left side of the header addresses the system, sub-system, or aspect of the system relative to 
the analysis at hand, along with the current system lifecycle phase and the key responsible 
parties. Here, it is important to be specific in designating the real-world focus of the analysis. 
Simply stating a particular analysis relates to, for example, the Space Shuttle or the MSL, is too 
vague. Some details of the RWS worth noting are the current system lifecycle phase, the sub- 
system or component of the RWS under analysis, and the aspect (a particular part or feature) of 
the system under analysis as described below: 

a. Current system lifecycle phase as designated by the project management phases 
defined in NPR 7120.5: The expectations of an analysis most likely change as a project matures. 
For example, the RWS operational performance analysis in the Concept and Technology 
Development Phase of a project is most likely much less accurate than the same analysis in the 
Operations and Sustainment Phase. The defined NASA project management phases are: 

( 1 ) Pre-phase A - Concept Studies. 

(2) Phase A - Concept and Technology Development. 

(3) Phase B - Preliminary Design and Technology Completion. 
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(4) Phase C - Final Design and Fabrication. 

(5) Phase D - System Assembly, Integration and Test, and Launch. 

(6) Phase E - Operations and Sustainment. 

(7) Phase F - Closeout. 

b. Sub-system, element, or aspect of system under analysis: One item very important to 
a more complete understanding of a problem and the associated analysis is its decomposition, 
which identifies the focus of the analysis, e.g., sub-system or component, in the context of the 
greater system of which it is a part. (See examples in section 5. 2. 1.2 of this Handbook.) Some 
typical examples of an engineering sub-system (element, portion, or aspect) of the system under 
analysis are: 

(1) Structures. 

(2) Mechanisms. 

(3) Fluids. 

(4) Thermal Management. 

(5) Electrical Power. 

c. Responsibility Chain: Additionally, understanding the roles of program/project 
management and the Technical Authority for a program or project should be recognized and is 
discussed in NASA-STD-7009 and in NPR 7120.5. NPD 1000.0, Governance and Strategic 
Management Handbook, also addresses the check-and-balance structure in the NASA 
organization with the separation of Programmatic Authorities from the Technical Authorities. 
Specifically, section 4.1 of NASA-STD-7009 states: 

“Program and project management have the responsibility to identify and 
document the parties responsible for complying with the requirements in this 
standard.” 

The Technical Authority, however, is largely responsible for accepting compliance with or 
deviations from the requirements of NASA-STD-7009. A depiction of these interrelationships 
and the required dual chain of responsibility are shown in figure 5, Relationships in the 
Responsibility Chain. 
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Figure 5 — Relationships in the Responsibility Chain 


5.1.2 Right Side of Header 

On the right side of the header, information relative to the M&S used in the current analysis is 
documented, including the M&S (or M&S system) designation, the M&S responsible party, the 
subject phase of the analysis, the Worksheet completion date, and whether NASA-STD-7009 is 
required by the M&S Risk Assessment in Appendix A of the Standard. 

Some nominal examples for the analysis phase by the M&S are: 

a. Production (or Manufacturing). 

b. Ground Operations. 

c. Launch Count Down or Pre-Flight. 

d. Ascent or Take-Off. 

e. Mission Operations (may include “on orbit” for spacecraft or “en-route” for aircraft). 

f. Entry (for spacecraft). Descent, and Landing. 

Many alternatives, detailed sub-phases, or off-nominal conditions, such as aborts, are also 
possible. Additionally, the environment may be the subject of the analysis. 

The M&S Risk Assessment in Appendix A of NASA-STD-7009 results in a categorization of the 
potential risk associated with an impending analysis. The result is a placement of potential risk 
on a matrix in critical (red), moderate (yellow), or relatively low risk (green) areas; however, the 
outcome of the assessment, in discussion with management, decision makers, and Technical 
Authorities, is a determination of whether NASA-STD-7009 is required. While the example 
shown in Appendix A of NASA-STD-7009 is a 4x5 matrix, the program/project determines the 
requisite dimensions or tool. Current formal NASA documentation does not specify the 
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dimensions of a risk matrix, but a 5x5 matrix appears to be most common. (See figure 6, M&S 
Influence-Decision Consequence (5 x 5) Risk Matrix.) 


M&S 

Results 

Influence 

5: Controlling 

(G) 

(Y) 




4: Significant 

(G) 

(Y) 

(Y) 



3: Moderate 

(G) 

(Y) 

(Y) 

(Y) 


2: Minor 

(G) 

(G) 

(Y) 

(Y) 


1 : Negligible 

(G) 

(G) 

(G) 

(G) 

(Y) 



V: Negligible 

IV: Marginal 

III: Moderate 

II: Critical 

I: Catastrophic 



Decision Consequence 


Figure 6 — M&S Influence-Decision Consequence (5 x 5) Risk Matrix 


For a technical decision where engineering judgment suffices, if M&S-based analysis results are 
used at all in supporting that decision, the influence of M&S-based analysis results may be 
considered minor or even negligible. In such a case, the M&S would be identified as non-critical. 
This Handbook and Appendix A in NASA-STD-7009 are used to help assess the criticality of 
M&S using a risk-based approach. 

The concept of Risk-Informed Safety Case (RISC), introduced in Volume 1 of the NASA 
System Safety Handbook (NASA/SP-201 0-580, November 201 1), is grounded on marshaling 
evidence to substantiate claims that the safety objectives of the system have been met or are on 
track toward being met. The M&S results often represent key evidence to substantiate safety 
claims. 


5.2 Section 1 - System and Analysis Frameworks 

The purpose of the first section of the Worksheet is to provide the basis and context for the 
analysis results. The context for any decision or analysis result is the RWS; therefore, the portion 
of the RWS under analysis, including the boundaries, need to be understood first, along with the 
problem definition. Second, the basis for the M&S-based analysis results, which includes the 
basic structure of the M&S, the abstractions and assumptions, the analytical boundaries of the 
M&S system, and limits of operation, need to be understood. With this understanding of the 
analytical context of the RWS and the M&S, an assessment of the correspondence between them 
is possible. Only after this assessment is accomplished should the results of the analysis be 
considered (section 2 of the Worksheet). 

5.2.1 RWS and Problem 

The first and last consideration for any M&S is the RWS, either as it exists or as it is intended to 
exist. Without reference to the RWS, M&S reduces to an academic exercise. Therefore, a clear 
understanding of the analysis focus with the M&S is the starting point. 

Clearly defining the RWS is the first step in the analysis of a problem, and it is from this 
definition that the context of the analysis is understood. The term system can be taken in many 
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ways, and its use in the vernacular can take on various meanings in the same conversation, so a 
clear understanding of the context of its use is required. For the purposes of this document, the 
RWS is the system, or portion thereof, under analysis. Typically, the environmental influences 
on the RWS are also included in the definition of the model. 

This portion of the Worksheet (figure 7, Worksheet Item: Real World System/Problem) solicits 
key aspects of the RWS included in the analysis, along with significant external or 
environmental influences and the specific problem addressed. 


Item | V | Result ICommen* 


System & Analysis Frameworks 


Real World 
System / Problem 

What is the real world system? 




What is its environment? 




What is the problenVdccision? 





Figure 7 — Worksheet Item: Real World System/Problem 


The intention of this Worksheet item is to obtain a clear and complete understanding of the RWS 
under analysis: 

a. The overall RWS and the focus area of the analysis should be annotated in the header 
portion of the Worksheet (figure 4). 

b. The environmental realm, or operating domain, that is the focus of the analysis. 

c. The problem of interest for this system, sub-system, or aspect of the system in the 
specific environment or during a specific operational phase that is the focus of the current 
analysis. 

An architectural diagram of the RWS focus area, which should include the significant influences 
from interfacing or environmental elements, can be instructive. 

5.2.1. 1 Explanations 

5.2.1. 1.1 Defining the RWS 

In understanding the real system and real environment, it is often useful to separate the RWS 
modeled from where, how, and under what conditions it is performing the operation. Classifying 
the system elements also helps visualize the modeled problem. System descriptions and details 
may include a variety of characteristics, e.g., statistics, properties, inherent variabilities, 
sensitivities, historical data, design maturity, and uncertainties. 

Engineered systems can be modeled at any point in the design stage, while complex natural 
systems, e.g., non-engineered physical systems and human systems, can be modeled only to the 
degree that knowledge or appropriately representative hypotheses are available to represent their 
functional response. If design maturity or system knowledge is low, then sensitivities that take 
into account uncertainties and design choices need to be addressed and inventoried. 
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Note that low design maturity or idealized representations of the functional responses of complex 
natural systems may result in a low M&S-based analysis credibility rating. This underscores the 
fact that the credibility of analysis results may have little association with the model itself. 

The RWS environment refers to external elements that affect it in a significant way, 
e.g., vacuum, temperature, dust, torque, or gravitational constant. A clear and complete 
understanding of the environment is a key part of understanding the RWS. 


5.2. 1.1.2 Problem Statement 

A clear statement of problem, decision, or technical issue should be documented. It is often 
stated as a question or an issue description with the appropriate context but without any 
methodological prescription or inferred solution. This is the question to be answered using the 
model, and it should stem from the needs of the required decision. The model, simulation 
analysis, and information in the resulting recommendations should be tailored to this, not vice 
versa. 

The problem statement and domain are best formulated at the time the modeling/analysis task is 
given, preferably through a negotiation between the modeler/analyst and key stakeholder(s). 

Key questions to help in the problem statement formulation are: 

a. What is the decision to be made? 

b. What part of the decision will stand on the model and analysis? 

c. What outputs and conclusions are necessary to support the decision? 

5.2. 1.2 Examples 


From any complete systems perspective, the following need to be known: 


What Must be Known 

a. The overall RWS 

b. The problem focus area, e.g., the sub- 
system^) involved 

c. The environmental realm or operating 
domain that is the focus of the 
analysis 

d. The problem of interest for the system 
in the specific environment or during 
a specific operational phase 


Example 

a. The Space Shuttle - Orbiter 

b. Orbiter Thermal Protection System, e.g., tile 

c. Shuttle ascent from launch to T +9 minutes 
in clear and calm weather 


d. The induced stress or fractures on the 
Orbiter tile from impact 


With this context, data from the analysis about the level of induced stress or potential of fractures 
on the Orbiter tiles from impact can be used to make decisions with regard to flight safety. 
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These basic descriptions of this particular real-world situation provide a clear understanding of 
the problem boundaries as basis for a recommended solution (decision). 

There are many methods of depicting the RWS to better understand its relationship to the 
component parts. Architectural, exploded, and relationship diagrams are useful in pinpointing the 
specific area of an RWS that is modeled and analyzed (figure 8, Exploded Diagram of the Space 
Shuttle Solid Rocket Booster, and figure 9, Relational Diagram of the Space Shuttle External 
Tank). 
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Figure 8 — Exploded Diagram of the Space Shuttle Solid Rocket Booster 
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Figure 9 — Relational Diagram of the Space Shuttle External Tank 
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A conceptual model (defined in section 3.2 of this Handbook) is useful in showing the functional 
relationship of a system’s parts. For example, reaction wheels, momentum wheels, and control 
moment gyros are devices used to provide attitude control for remote sensing missions. When 
the rotors in these devices are spinning, inhomogeneity in mass distribution produces centrifugal 
forces and moments (similar to those in an unbalanced automobile tire or a load of laundry in a 
washing machine) that will affect pointing performance and result in blurred images. Arbitrary 
inhomogeneous mass distributions in the rotors may be modeled as discrete, lumped masses 
attached to ideal cylinders, as illustrated in figure 10, Conceptual Model - Free Body Design. 


Static Imbalance Dynamic Imbalance 



Figure 10 — Conceptual Model - Free Body Diagram 


A control system designer may need such a model to address many questions related to dynamic 
performance of a satellite. For example: 

a. What imbalance specifications (coefficients U s and U<j in the diagram) need to be 
levied on the reaction wheel supplier to meet satellite pointing requirements? 

b. What is the effect upon satellite pointing performance for a given reaction wheel with 
measured imbalance coefficients? 

If the analysis problem is more process-based, a conceptual flow diagram is useful in showing 
the relationships of a system’s component processes and resources (figure 1 1 , Conceptual Flow 
Diagram of a Process). 


Facility ^Resource 
/ / 
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With such a model, an operations planner can analyze a system’s ability to meet launch rate 
expectations and make decisions with respect to component supplier arrival rates, resource 
(facility and transporter) quantities, or work shift and depot cycle policies in an attempt to 
improve the system’s processing performance. 

5.2.2 Model/Abstractions/ Assumptions 

This Worksheet item (figure 12, Worksheet Item: Model/ Abstractions/Assumptions) is used to 
understand the M&S used in a given analysis. Models are developed from abstractions of an 
actual or proposed RWS implemented using the algorithmic logic of a conceptual model. (See 
section 4.2.2 of this Handbook.) Assumptions and abstractions focus model development for a 
particular purpose and should not have adverse influence on intended analysis outcomes. Logic 
and algorithms, which define implementation of the abstraction, including assumptions and 
limitations, are provided through the conceptual model. Items known to be missing from the 
M&S or solution and the modeling level of detail, i.e., fidelity, should be documented, e.g., the 
level of accuracy/precision in a geometric model and the time-step granularity in a process 
model. In addition, any limitation of the execution environment to represent the RWS fully, from 
the computational hardware, virtual machine, or software applications used to implement and run 
the model or simulation, should be understood and documented. 

Item | •/ 1 Result [Comments 


System & Analysis Frameworks 


Model / Abstractions / 
.Assumptions 

What is the M&S approach? 




What's included in the M&S, including 
model environment influences? 




Is there anything significant to this analysis 
not included in the M&S or scenarios? 




What assumptions & abstractions are 
included in the M&S and Analysis? 





Figure 12 — Worksheet Item: Model/Abstractions/Assumptions 


This Worksheet item is characterized by four general issues: 

a. The M&S approach, methods, and architecture. 

b. What is included in the M&S, including model environment influences. 

c. Significant omissions to this M&S or scenarios for the analysis. 

d. Significant assumptions and abstractions for the M&S, scenarios, and analysis. 

The following items illustrate the results and comments a reviewer might make in the Worksheet 
after review of a model or simulation: 

a. What are the M&S approach, methods, and architecture? 

Result: Acceptable. 

Comments: A finite element transient solution of launch dynamics. The simulation 
behavior is of sufficient granularity to address the analysis problem; a large set of 
conditions can be screened and problem areas noted for further detailed assessment. 
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b. What is included in the M&S, including model environment influences? 

Result: Acceptable. 

Comments: Computational environment and model abstraction and conceptualization 
explained. All important aspects of the RWS are included with no shortcomings seen 
at this time. 

c. Is there anything significant to this analysis not included in the M&S or scenarios 
(definition in section 3.2 of the Handbook)? 

Result: Unacceptable. 

Comments: Model failed to capture significance of temperature-dependent material 
properties on deflection control. 

d. What assumptions and abstractions are included in the M&S and analysis? 

Result: Acceptable. 

Comments: Physics of the situation in the model are adequately formulated. 
Conceptual model is sufficiently documented. 

5.2.2.1 Explanations 

The questions provided for this Worksheet item are provided to ensure a complete explanation of 
the M&S and should consider the following aspects: 

a. M&S approach, methods, and architecture 

(1) The modeling methods used, e.g., mathematical, stochastic/probabilistic, 
geometry, discrete event, relational, behavioral, physical, agent-based, human in 
the loop, and hardware in the loop). 

(2) The level of fidelity/detail represented in the approach/model, as characterized by 
typical expected/desired results accuracy. 

(3) The architecture (or diagram) of the M&S system, to include: 

A. A high-level architectural diagram (or conceptual model) of the M&S system 
with major components and their respective interfaces, which provides an 
understanding of the solution objective. (See figure 13, Flow Diagram of 
Space Shuttle Launch Site Process; figure 14, Conceptual Model of Space 
Shuttle Launch Site Process; and figure 1 5, M&S Architectural Diagram, in 
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section 5.2.2.2.1 of this Handbook.) There should be a relationship to the real- 
world problem with acceptable abstraction, including: 

i. Physical world environmental influences, e.g., pressure, thermal, 
electromagnetic. 

ii. Data I/O requirements. 

iii. Interoperability requirements with other M&S. 

iv. Solution accuracy considerations. 

B. If the model is computer-based, define and/or diagram the computational 
architecture with key hardware and software components, including 
dependencies or restrictions and stand-alone or distributed platform and 
related details or issues, e.g., machine run-time speed, capacity, bandwidth 
accuracy, computing centralization or distribution, and use of homogeneous or 
heterogeneous computational elements. 

(4) The types of problems the M&S is intended to support, e.g., training, force 
structure analysis, command and control, experimentation, component design, 
system analysis, or analysis of alternatives. 

(5) Key data features (such as input data and output data) should be noted: 

A. For input data, identify the information required to populate and execute the 
M&S, including input data sets, hard-wired data, i.e., constants, environmental 
data, and operational data. Provide descriptive metadata, metrics, and 
authoritative or approved sources for all data. 

B. For output data, identify the results from an M&S run, including a definition, 
the unit of measure, and the range of values for each data item. 

b. Inclusions in the M&S, including environment effects 

( 1 ) State how the model is abstracted into a problem of quantifiable solution. 

(2) Identify model components in the conceptual model, and describe how the model 
solves the problem abstraction (MIL-STD-3022, Standard Practice 
Documentation of Verification, Validation, and Accreditation (VV&A) Models 
and Simulations). Abstraction provides a generalization of the problem, which 
reduces the information content to a more easily implemented solution, with focus 
toward a particular relevant purpose. 
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c. Significant omissions to the M&S or analysis, including scenario completeness. 

Include limiting factors and constraints on the solution, along with known 
omissions of significant features of the RWS or characteristics of the problem. 

d. Assumptions and abstractions of the M&S and analysis. 

( 1 ) Assumptions and abstractions should be noted along with (or in) the conceptual 
model and/or model specification. 

(2) Analysis bounds and M&S limits of operation should be quantified and 
maintained. 

5.2.2.2 Examples 

5.2.2.2.1 Space Shuttle Processing Model Example 

An example of a Space Shuttle process flow at the Kennedy Space Center (KSC) is shown in 
figure 13. 
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Space Shuttle Hardware Flow 
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Figure 13 — Flow Diagram of Space Shuttle Launch Site Process 


From this diagram, discussions with SMEs and a review of program documentation, a more 
complete conceptual model was developed (figure 14). 
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Figure 14 — Conceptual Model of Space Shuttle Launch Site Process 
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From the conceptual model, the computational model was developed using a single computer 
(Microsoft Windows®; no special requirements other than memory) and three separate software 
packages: a COTS discrete event simulation (DES) application, a custom-built user interface, 
and a COTS spreadsheet application. The user interface was used to define the set of scenarios 
with hundreds of parameters for system analysis, along with manifest definition data from the 
spreadsheet. The COTS DES application used these deterministic and stochastic inputs to run the 
model through a defined number of replications and produce statistical performance data of the 
RWS. In the architectural diagram (figure 15), the data flow between the three applications is 
seen. 
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Figure 15 — M&S Architectural Diagram 


5.2.2.2.2 Object Definition Conceptual Model Example 

Figure 16, Conceptual Modeling with COTS Software, shows the utility of conceptual model 
development using commercial software. (See DoD RPG Special Project Concept Model 
Development and Validation.) The multiple views shown provide a mechanism to define and 
interrogate conceptual model characteristics through multiple data and functionality 
representations. Hierarchical data trees and associated graphical views of class, component, and 
use case data are illustrated. Additional techniques for performing comparative analysis between 
the conceptual data model for new systems and the physical data models represented in 
applications, databases, and related systems are found in Hagan & Walker, 2009. 
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Figure 16 — Conceptual Modeling with COTS Software 

DoD documentation and directives (MIL-STD-3022 and DoD Modeling and Simulation (M&S) 
Glossary) provide excellent process guidelines in documenting M&S activities, and Appendix C 
of MIL-STD-3022 could be utilized as a tailorable template to incorporate into M&S activities. 

5.2.3 System - Model Match 

This Worksheet item flows naturally from sections 5.2.1 and 5.2.2 of this Handbook. With 
separate understandings of the RWS and the M&S, the intent now is to assess how well the 
M&S, as it is intended to be used, matches the RWS. It is understood that in novel cases, i.e., 
new or one-of-a-kind missions, the similarity between the RWS and the M&S may not be 
completely determinable before the first flight. At times, the appropriate data are not even 
collected during a given mission to validate an existing model. However, some basis, e.g., an 
analogous referent, should be available for accepting the model as appropriate to the analysis. In 
such cases, an understanding of what is known and unknown about the M&S and the RWS is 
used as a basis for acceptance. 

a. What makes the M&S a good representation (or portion-of-interest) of the RWS? 

(1) Relevant portions or aspects of the RWS are included in the model. 

(2) The accuracy and fidelity of the M&S are to be adequate for representing the 
RWS. 
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(3) Model abstractions, assumptions, or functions different from the RWS are to be 
identified and assessed. 

(4) The validated operating domain of the model is to be assessed with respect to the 
targeted operating domain of the RWS, including relevant environmental 
characteristics, with differences identified and assessed. 

b. It is important to understand how the M&S directly produces the results necessary for 
the analysis. 

(1) Is the intended use of the M&S directly applicable to the RWS problem? 

(2) Are the M&S results a direct answer to the RWS problem? 

(3) Is post-processing of the M&S output required? 

(4) If M&S results post-processing is required, then: 

A. Of what does it consist? 

B. Is it part of V&V? 

C. How is it documented? 


The above information may be provided for this Worksheet item (figure 1 7, Worksheet Item: 
System - Model Match), with annotations, explanations, or references to other documentation 
given in the Comments field: 


Item Iv'lResult |Comments 

System & Analysis Frameworks 

System - Model 
Match 

How well does this M&S represent the 
Real World System / Problem at hand? 




How well does this M&S produce the 
results necessary for this analysis? 





Figure 17 — Worksheet Item: System - Model Match 


5.2.3.1 Explanations 

5.2.3.1.1 What makes the M&S a Good Representation of the RWS (or Portion Thereof)? 

All models are implicitly limited. What is inferred from this statement is that models only behave 
as designed and do not capture every behavior exactly like an RWS, act like all possible 
instances of an RWS, or encompass all aspects of an RWS. While the RWS may currently exist 
or may only be conceived, a model is a system representation. An understanding of how well the 
M&S matches the RWS is imperative to ensure the adequacy of system representation and the 
specific problem under consideration. 

Validation is the primary activity during M&S development to understand analytically how well 
a model represents an RWS. A typical situation is illustrated in figure 18, Notional Illustration of 
a Validation Domain, for the case of two parameters of the system and/or the environment. The 
solid circles represent points in the two-dimensional (2-D) parameter space at which validation 
referent data are available. The solid line represents the envelope of the validation points, which 
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encloses the validation domain, i.e., the region in which the M&S results have favorable 
agreement with the referent data. The dashed lines represent the boundary in which the key 
model assumptions hold, and reasonable results are expected. In many cases, the referent data do 
not cover the full region in which the limits of model assumptions are satisfied. The color 
symbols represent points for which the M&S results are inside the validation domain (green); 
outside the validation domain but still within the limits of assumptions (yellow); and outside the 
limits of assumptions (red). The validation activity should clearly define and document these 
boundaries. 


Outside Assumptions 



Note that any point within the validation domain that does not exactly match a validation point is 
an interpolation. While interpolated results are generally considered acceptable, the nearness to 
exact validation points should be considered, as well as whether linear or curvilinear 
interpolation is warranted. On the other hand, any results outside the validation domain are 
considered extrapolations and much more caution is warranted. Again, the nearness to any exact 
point can be considered, but the behavior of the system anywhere outside the validation domain 
is really unknown. 

When a model has been developed and validated for a particular RWS, then the validation 
boundaries are sufficient to determine the system-model match. However, when M&S are 
developed as a general solution to a class of RWSs, then the validation bounds need to be 
reassessed for each additional RWS to which the model is applied. Alternatively, systems are 
developed and accepted to operate within specified operational and environmental constraints. If 
an analysis is required with a model previously validated for a system but with the system 
operating outside previously specified margins, then a re-evaluation of the adequacy of the 
model is necessary. If the re-evaluation shows the M&S to be inadequate, then model 
modifications and/or additional validation are required. 

The preceding discussion focused on the validation bounds. Some of the same considerations 
apply to verification, although the model verification is not as strongly tied to a particular RWS 
as is its validation. 
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There are a few situations in which the adequacy of a model to represent the RWS comes into 
question: 

a. When a general-purpose model is applied to a particular system, situation, or 
problem. General-purpose models are created to address a wide range of problems. The intended 
use of such models should be considered carefully to ensure such a model is adequate to the 
situation to which it is applied, including the domain of validation, limitations of model 
assumptions, accuracy, precision, and producing results applicable to the problem it is 
addressing. In contrast, customized models are built to address a specific problem and should 
more directly comply with its needs. 

b. When an analysis is required of a system operating outside its nonrial limits. If an 
M&S is constructed to represent a specific system, the M&S is most likely validated only for the 
normal operating conditions for that system. Analyses of the system outside those normal limits 
of operations are extrapolations, which are not, by definition, within the bounds of validation. 
Therefore, the results of the analysis must be considered carefully and be accompanied with a 
placard. (Refer to the example in section 5. 3. 3. 2 below). 

c. When an M&S is used to analyze a problem for which the limits of assumptions are 
exceeded. If the limits of assumptions are exceeded, great care must be taken with the suggested 
implications and with full knowledge of the risks involved. 

Note: Both NASA-STD-7009 and this Handbook focus on M&S. The most credible use of any 
M&S is within the domain of V&V. Understanding the V&V domain in relation to the targeted 
operating domain of the RWS is crucial to understanding the M&S-based analysis. If the M&S 
use is extrapolated beyond the domain of V&V, then the analysis is to be accompanied by the 
appropriate caveat. 

5.2.3. 1.2 How Well Does the M&S Produce the Necessary Analysis Results? 

An M&S is developed to address a particular system or problem type. If a model represents a 
system well but does not produce the required statistics or figures of merit on which to base a 
conclusion, then some adjustment or post-run manipulation of the data is required. Ideally, this is 
considered in the V&V phase of the M&S activity; however, tailoring of an established M&S to 
a specific purpose is also possible. In such cases, the M&S may not directly produce a needed 
result, but one may be derived. This process of derivation is to be carefully considered and 
documented. 

5.2.3.2 Examples 

Note: Examples will be developed and provided in a later revision. 
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5.3 Section 2 - M&S-based Analysis Results & Caveats 

With the understanding of the RWS and the M&S and an acceptance of their correspondence, a 
look at the analysis results is appropriate, with the reminder that the M&S and the results 
produced are approximations with uncertainty. Any further qualifying statements surrounding 
the analysis should be included as a caveat to the analysis. 

5.3.1 Best Estimate 

The notion of a best estimate of results may be deceptively simple; however, it is critically 
important to remember that all M&S results are estimates and, therefore, contain uncertainty of a 
given system’s response and not necessarily the exact response to expect from the RWS. It is 
also important to understand that, generally, the direct output of the M&S is itself not the final 
answer. A typical analysis requires multiple simulation runs, followed by some post-processing, 
possibly including statistical or non-statistical representation of output data. This can and often 
does involve the use of advanced statistical methods. 

In at least some cases, the analyst may provide multiple answers to the problem; these answers 
are dependent on specific assumptions affecting the M&S from model form to model parameters 
to simulation run-time parameters. Each result is effectively a conditional best estimate. 

The important questions to ask when presented with a best estimate include: 

a. What definition of best estimate was used by the analyst? 

(1) Mean, median, mode, maximum likelihood? 

(2) Were higher order statistical measures considered? 

(3) Were outliers removed? 

b. Is there agreement on the problem definition? 

In question a(3) above, an outlier is simply defined as a data point in a set that lies outside the 
expected range of values. In practice, such a data point should be studied carefully and only 
removed if it is clearly an aberrant piece of data. It is also possible that a suspected outlier is a 
valid data point that happens to represent extreme values of the data set, which is detrimental if 
eliminated from an analysis. Additionally, in some studies with small data sets, eliminating any 
piece of data can significantly affect the representative statistics. 

For this Worksheet item (figure 19, Worksheet Item: Estimate), the single valued result of an 
analysis may be put in the Result column, with qualifying statements, notes, or references to 
analysis documentation in the Comments column. 
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Item | S | Result |Comments 

M&S-based Analysis Results & Caveats 

Estimate 

What are the best-estimate results provided 
by the analysis? 




How well do the analysis results address the 
problem statement? 




Figure 19 — V 

worksheet Item: Estimate 


5.3.1. 1 Explanations 


Possibly the most common statement of a measurement or prediction includes both the best 
estimate and the uncertainty and is expressed as: 

Measurement or Prediction = Best Estimate ± Uncertainty 

It is not always the case that the term best estimate is synonymous with commonly applied 
statistical measure, e.g., mean, median, maximum likelihood. The problem statement may be 
looking for extremes in expected outcome rather than a mid-point. In such cases, the results of a 
bounding analysis are driven to the extremes, i.e., are “no less than” or “no greater than” values, 
which provide neither two-sided error bars nor any kind of result distribution. A typical example 
involves predicting upper/lower temperature bounds for electronics. Many parameters in a 
thermal M&S are adjusted to provide hot or cold extreme predictions, which remove most or all 
of the aleatory uncertainties, i.e., naturally occurring randomness, in the estimate. 

One consideration in the case of linked M&S, i.e., when the output from one M&S provides 
inputs to a second M&S, is that an M&S (or system of M&S) may serve multiple purposes. For 
example, the same M&S used to predict upper/lower bounds on temperatures may also be used 
to predict temperature changes over time that impact alignment stability of optics or other 
hardware. In the latter case, the desired best estimate may be the nominal value. It does no good 
for the structural M&S to be used with nominal parameters if the thermal M&S does not, as the 
end-to-end results will not reflect nominal performance. It is recommended to verify that the 
parameters of the thermal M&S were adjusted for each appropriate case, which is also 
considered under the CAS Input Pedigree factor. 

5.3. 1.2 Examples 

The estimate for any given M&S-based analysis can take many forms. The following two 
examples will be further enhanced in sections 5. 3.2.2 and 5.4.5.4 of this Handbook. 

a. Example 1 - An analysis of the operations, integration, and launch processing of an 
operational launch vehicle in the middle of its design phase showed an average possibility of 
3.96 launches per year with certain constraints. 

b. Example 2 - The timeline analysis for a single process yielded the time-length of the 
process. From the 200 samples points obtained, the process average time was 2.00 days. 
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5.3.2 Uncertainty in Estimate 

The Uncertainty in Estimate item is a high-level, aggregate view of uncertainty as it is 
manifested in the results of the M&S-based analysis. The details of the contributing sources of 
uncertainty in the M&S, the input to the M&S, and the propagation through a simulation analysis 
are discussed in the CAS - Results Uncertainty item (section 5.4.5 of this Handbook). 

As the analysis results are based on a model of the real system and its environment with 
concomitant assumptions, approximations, estimates, and other uncertainties, it is inappropriate 
and possibly misleading to present the analysis outcome as a single definitive result without 
qualification. 

Important questions to ask for this Worksheet item (figure 20, Worksheet Item: Uncertainty in 
Estimate) include: 

a. What are the magnitudes of the uncertainties in the analysis results? 

b. Are the uncertainties understandable and reasonable? 

c. How does the uncertainty influence the decision at hand? 

d. How does the uncertainty influence the risk associated with the decision at hand? 


Item |/|Result |Comments 

M&S-based Analysis Results & Caveats 

Uncertainty 
in Estimate 

What are the magnitudes of the uncertainties 
in the results of this analysis? 




Figure 20 — Worksl 

beet Item: Uncertainty in Estimate 


The Result column can be used to document the magnitude of the aggregate or relative 
understanding of the results uncertainty, e.g., the percentage variation in the results, the standard 
deviation, half-width, statistical confidence interval, or other interval bound. The Comments 
column can be used for further clarifications or for references to supporting documentation. 

5.3.2. 1 Explanations 

5.3.2.1.1 Reporting Results Uncertainty 

The associated risk of accepting an analysis result increases if a discussion of uncertainty is not 
included. The topic of uncertainty is complex and difficult; however, the discussion helps to 
preclude misunderstandings, unjustified expectations, or conclusions that are either overly 
optimistic or overly pessimistic. 

Uncertainty is most often described in statistical or probabilistic terms, e.g., the uncertainty of a 
measurement or the probability density function (pdf) of an inherently variable environmental 
parameter. However, other mathematical descriptions are also used, including interval bounds 
related to lack of knowledge, e.g., the reaction rate of a chemical reaction for which 
measurements are unavailable. (For a discussion of alternative uncertainty descriptions, see 
SAND2003-3769, Verification, Validation, and Predictive Capability in Computational 
Engineering and Physics.) For simplicity, the following discussion is confined to statistical or 
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probabilistic descriptions. When alternative descriptions are used, the suggestions described 
below should be adjusted accordingly. 

The communication of uncertainty is accomplished in many ways. Common methods include a 
mean ± calculated error bands, e.g., a 5 th and 95 th percentile and/or confidence bands, and 
graphical methods, from box-plots to uncertainty distributions shown within a physical context. 
Graphical methods help visualize the span or range of the result from an analysis. The numerical 
uncertainty around an analytical result may be shown as follows: 

• Estimate ± Standard Deviation. 

• Estimate ± Half Width. 

• Confidence Interval. 

Raw sample standard deviations, histograms, and coefficients of variance relay incomplete 
information and are not recommended as full descriptions of uncertainty. Information on sample 
size and estimated distribution, e.g., normal, Weibull, nonparametric, skewed, or unknown, 
improve the utility of the spread information by giving information on the quality and 
quantitative bias of the estimate. 

The following key points should be clearly understood and reasonable, i.e., make sense in the 
context of the RWS: 

a. The best estimate from a given analysis is most likely a deterministic or single-valued 
answer, unless stated otherwise, e.g., the minimum/maximum probable value is or the best/worst 
case possibility is. It should be clear what type of result is given. 

b. When the uncertainty for a given result is provided, it should be understandable and 
reasonable in the context of the RWS/problem. For instance: 

(1) The uncertainty bounds should be physically possible, e.g., negative pressures or 
times are signals of incorrect calculation or assumptions. These estimates should 
be considered in context with the RWS. 

(2) If the reported mean or median value is incorrect and the real system performs in 
a reasonably conservative part of the uncertainty bounds, then would this cause a 
problem? 

(3) Does the resulting decision change when moving within the upper and lower 
bounds of the range of uncertainty? 

c. When uncertainties are provided, does the decision risk change significantly with 
respect to the upper and lower bounds of that range of uncertainty? 

d. Are the significant sources of uncertainty understood, as discussed in section 5.4.5 of 
this Handbook? 
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Qualitative estimates of uncertainty can be as useful as quantitative ones, though they may be 
more difficult to use to specify a resulting risk. The magnitude of the effect is most likely also 
qualitative. 


5.3.2.1.2 Cautions 

There are many ways of depicting uncertainty in data graphically, with possible adjustments to 
suit the needs of a particular situation. A box-plot (or box-and- whisker plot) is one such example 
for displaying the statistics of an associated data set. Key points on the statistical display of data 
should be clearly labeled and their meaning clearly conveyed. 

Uncertainty estimates that appear too optimistic, e.g., little to no variation or 3-a estimates, or 
bounds drawn based on the range of data should be carefully questioned and may indicate an 
incomplete uncertainty assessment. 

Uncertainty estimates that are more constricting than necessary may inappropriately drive 
program/project decisions. 

When using a given confidence interval to inform decisions, be aware there is uncertainty about 
the confidence interval as well. 

5.3.2.2 Examples 

Further analysis from the first example in section 5. 3. 1.2 of this Handbook for launch vehicle 
operations, integration, and launch processing produced an understanding of the uncertainty 
around the estimate. Figure 21, Cumulative Probability Distribution of Analysis Results, shows 
there was an 82 percent probability of launching four or more times per year when trying to 
maximize the launch rate; however, there was also an 18 percent probability of launching only 
two or three times per year. 


Cumulative Probability of Achieving X or More Launches 

91-Day Mission Request; Pad Scrubs and Rollbacks On; 



5 4 3 2 1 0 

Number of Launches mam Percentage ^ 


Figure 21 — Cumulative Probability Distribution of Analysis Results 
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To answer questions a through d from section 5.3.2 (above) for this example: 

a. What are the magnitudes of the uncertainties in the analysis results? 

The magnitude of the uncertainty for these results is essentially ±one launches per 
year with the specific probabilities shown in the graph and discussed previously. 

b. Are the uncertainties understandable and reasonable? 

Further understanding of the uncertainties requires a more detailed look at the M&S 
to determine why the number of launches is constrained. Either long process times or 
constrained resources are possible causes. 

c. How does the uncertainty influence the decision at hand? 

This uncertainty shows the inability of the modeled system to reliably maintain a 
constant launch rate greater than two. If program objectives require more than that, 
then changes to the operational system are required. 

d. How does the uncertainty influence the risk associated with the decision at hand? 

The uncertainties in the graph show more significant risk in reliably attaining more 
than three or four launches per year. 

5.3.3 Caveats 

For M&S-based analyses, the term caveat is defined as 

Modifying or cautionary information to consider when evaluating or interpreting the 
results of an M&S-based analysis. 

In a given M&S-based analysis, a caveat is information pertinent to the results presented that 
should be documented and provided as a caution to the recipient, e.g., decision maker. 

The reporting of caveats to an M&S-based analysis is initially stated in section 4.7 of 
NASA-STD-7009, with details called out in Requirement 4.8.1 of NASA-STD-7009. 

NASA-STD-7009 includes requirements and recommendations for reporting results to decision 
makers. When describing the philosophy behind credibility assessment and its role in the 
decision-making process, the notion of caveats is introduced. The operational concept of the 
CAS requires that the presentation of M&S-based analysis results to a decision maker include: 

• The best estimate of the results. 

• A statement on the uncertainty in the results. 

• The evaluation of the results on the CAS. 

• Any explicit caveats that accompany the results. 
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Examples of possible caveats are: 

a. Violations of M&S acceptance criteria, assumptions, or restrictions, e.g., limits of 
operation. 

b. Exclusions from the model that significantly impact the results, the uncertainties in 
the results, and the conclusions derived from those results. 

c. Errors and warnings that occur during an M&S-based analysis run. 

d. Unfavorable outcomes from the intended use and setup/execution assessments. 

e. Any waived requirements of NASA-STD-7009. 

f. Analysis coverage space is inadequate or limited, e.g., as in a limited design of 
experiments. 

g. Analysis focused on a specific design or vehicle configuration. 


The types of information to include in this Worksheet item (figure 22, Worksheet Item: Caveats) 
are caveats that either should be considered or cause a rejection of the presented results. 



Item 


|Comments 

M&S-based Analysis Results & Caveats 

Caveats 

What are the caveats to the analysis with 
this M&S? 





Figure 22 — Worksheet Item: Caveats 


5.3.3. 1 Explanations 

Each caveat should be assessed individually as to its impact on the analysis. Some details, 
events, cautions, or warnings to document as caveats may be: 

a. Errors and warnings that occur during an M&S-based analysis run: Understanding the 
source of the errors/wamings and why they occurred 

b. Violations of M&S acceptance criteria, assumptions, or restrictions: Not operating 
within the acceptance criteria for the M&S. A common practice is to identify qualitative and/or 
quantitative metrics, i.e., quality or goodness indicators for the M&S to be included in the 
acceptance criteria. An example from a structural finite element M&S (Bolognese, 2009) would 
be numerical tests (or flags) for indicators of an ill-conditioned stiffness matrix before the 
inversion of the matrix in a static analysis. Thresholds are set for a number of such scalar 
indicators. Analysis results may be accepted cautiously if one or more thresholds are exceeded 
slightly but generally are not in cases when any or all thresholds are exceeded to a significant 
degree. 
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c. Not adhering to the assumptions either built into the model or in the input to the 

model. 

d. Violations of limits of operation: In instances where thresholds are exceeded but the 
analysis should be annotated with appropriate caveats and cautions. 

e. Inadequate level of detail included in the model: Insufficient model detail discovered 
by the Worksheet items Model/Abstractions/Assumptions and System/Mode/Match should be 
listed in caveats. 

f. Analysis coverage space is inappropriate or limited: Insufficient analysis coverage 
discovered, i.e., by Worksheet item Validation, should be listed in caveats. 

g. Analysis focused on a specific design or vehicle configuration: Poor applicability of 
the analysis to the RWS as discovered by the Worksheet item System/Model/Match should be 
listed in caveats. 

A good practice for reviewing M&S results would be the inclusion of a line-by-line examination 
of the documented assumptions and abstractions to: 

• Determine if any of these were violated. 

• Assess the consequences of such violations on the accuracy or interpretation of the 
results. 

Furthermore, it is important to raise the same question about the possibility of undocumented or 
implicit assumptions. These situations might occur when off-the-shelf (OTS) M&S software is 
used for a variety of reasons, including: 

• • The (commercial) developers may not have documented their assumptions. 

• The operators and analysts may not be totally familiar with the software. 

• The operators and analysts may have used this code to the point where the underlying 
assumptions have effectively become implicit. 

5.3.3.2 Examples 

One method to ensure analysis caveats are noted adequately is to add placards to the results 
(figure 23, Example Placard). 
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CAUTION 


Analysis Performed 
Outside the 
Limits of Operation 


Figure 23 — Example Placard 

Another example is to note vehicle configuration differences between the M&S, defined analysis 
scenarios, and the RWS in the caveats. 

5.4 Section 3 - M&S Credibility Assessment 

The first two worksheet sections form the basis for making a decision with respect to the RWS. 
However, there are details yet to consider as to the credibility of the results of an M&S-based 
analysis. These details are included in section 4.7 and Appendix B of NASA-STD-7009 and in 
the section 3 of the Worksheet, which addresses key development, usage, and process aspects of 
an M&S activity. One of the early applications of this credibility assessment method was 
accomplished by the Orion Project and documented in NASA TM-201 1-215987, A Credibility 
Assessment Scoring (CAS) Process for Mission Risk Management. 

% 

a. There may be other key aspects to a particular type of M&S that are not included in 
this credibility assessment. Including them along with the credibility assessment defined in 
NASA-STD-7009 is acceptable and encouraged. The factors included in NASA-STD-7009’s 
credibility assessment are considered to be a minimal set for a majority of M&S. If, however, a 
factor is not considered relevant to a particular M&S, tailoring is permitted but only with the 
approval of the program/project Technical Authority. (See section 1.2.1 of NASA-STD-7009.) 

b. There is no correlation between compliance with the requirements of 
NASA-STD-7009 and the achievement of particular levels for the various factors in the CAS. 
Attaining the various levels of credibility relate to the technical aspects and are to be defined on 
a case-by-case basis. 

5.4.1 Overall Credibility 

The idea of M&S-based analysis credibility is necessarily complex and easily misconstrued, but 
it is a natural part of any decision-making process. As credibility cannot be measured directly, 
the methodology developed as part of NASA-STD-7009 formalizes this assessment with a 
minimum set of criteria contributing to M&S-based analysis credibility. 

The acceptable level for the overall credibility and contributing factors is determined by the 
program/project management in association with the Technical Authority, as appropriate for the 
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current state of the RWS and the M&S and the criticality of the decision being made. The 
expectation for analyses is that they improve as: 

• The system development matures. 

• Data become available from relevant phases of the program/project. 

• The M&S matures and is used. 

The assessment of overall credibility comes from a rollup of the credibility factor assessments 
for the M&S-based analysis and should include the following items: 

• A tabular or graphical display for all of the CAS factors. 

• The weighting used in calculating the each factor assessment, if the factor has a technical 
review sub-factor and the rationale for the weights. 

• The role of the person/team performing the credibility assessment in the development, 
operation, or analysis using the M&S. 

• A summary of the evidence and supporting rationale. (A reference to another document 
may suffice.) 

Information entered on the Worksheet is a synopsis of more detailed information from an 
assessment of the M&S and the analysis performed with it. Figure 24, Worksheet Item: Overall 
Credibility, shows how the information would appear using the data in the example of Appendix 
B of NASA-STD-7009. 



Item 

r ttrss 1 


M&S Credibility- Assessment (Development - Usage (Analysis) ■ 

- Process | 


Overall Credibility 

What are the overall results of the M&S 


Score: 1.7 

Who assessed the analysis? Jane Doe of the Assessment Team 


CredfciHty Assessment? 


Weighting Used? Yes 

Rationale summary? . . . (reference file name) 


Figure 24 — Worksheet Item: Overall Credibility 


5.4. 1.1 Explanations 

The overall credibility assessment of the M&S-based analysis is a synopsis of the individual 
factor ratings that follow in subsequent Worksheet items and CAS factor ratings. Reporting this 
information with an M&S-based analysis is required by Requirement 4.8.3 of NASA-STD-7009. 

This overall credibility information provides the single-valued credibility assessment. However, 
this assessment is to be supported by information from the individual factor assessments. 
Improvement of the overall credibility assessment is only possible through improvement of the 
contributing CAS factors, which may also include an improvement in technical review. When 
weighting is used in the assessment of the five factors with sub-factors, the weights and rationale 
should be provided as part of this overall assessment. A table, bar chart, or spider plot (radar 
plot) of the CAS aids in understanding the overall assessment. 
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5.4. 1.2 Examples 

An example of CAS reporting in accordance with Appendix B of NASA-STD-7009 with target 
threshold values is shown in figure 25, Graphical Methods for Reporting M&S Results 
Credibility. 

Verification 



a. Bar Chart b. Spider (Radar) Plot 

Figure 25 — Graphical Methods for Reporting M&S Results Credibility 


Note: It is not possible for an M&S to achieve Level 4 credibility without considerable effort in 
M&S development and use and without adequate data from the RWS. For example, many NASA 
scientific missions consist of a single flight vehicle. The only way to attain a Level 4 assessment 
for validation is by comparison with results from the actual RWS; therefore, any time before the 
mission, an assessment of Level 3 is the highest possible for validation. The purpose for such an 
assessment is to discuss the factors influencing the credibility of the analysis results. It is the 
decision maker’s responsibility, in conjunction with the Technical Authority, to ascertain the 
acceptability of this information. 

5.4.2 Verification 

The process of verification ensures the computational model (or simulation model) is correctly 
implemented. Verification does not ensure the M&S matches the RWS or addresses the problem 
of interest. The M&S can be considered verified when the following two conditions are satisfied: 

• The computational model meets its specifications. These software specifications start 
with the conceptual/mathematical model and include additional requirements for 
functions, e.g., user interfaces and data I/O. 

• All significant sources of numerical errors inherent in the software implementation are 
identified, quantified, and within assigned upper bounds. 

A review should examine the documented evidence relating to these two aspects of verification 
and address questions, including the following: 
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• What actions demonstrated the computational model functions exactly as intended, as 
specified by the conceptual model or other model requirements document? What were the 
results of these actions? 

• What process was used to quantify numerical errors resulting from the software 
algorithms, and what were the results? 

• What process was used to quantify numerical errors resulting from factors such as 
sampling or quantization, the step size chosen for the numerical integration of differential 
equations in a time-domain simulation, and the methods and intervals used for 
interpolation of model parameters; what were the results? 

The Results and Comments columns, shown in figure 26, Worksheet Item: Verification, could be 
used to note failures to satisfy all of the software specifications, the most significant potential 
sources of numerical error in the simulation, and at least a qualitative assessment as to the effects 
these differences may have upon predictions for the RWS. 


Item | S | Result | Comments 

M&S Credibility Assessment (Development - Usage (Analysis) - Processl 

Verification 

How (well) does the M&S implementation 
match the conceptual specification? 




Figure 26 — 

Worksheet Item: Verification 


5.4.2. 1 Explanations 

The term “verification” is generally accepted to refer to two related processes: code verification 
and calculation verification (or solution verification). These processes are designed to 
demonstrate that the software implementation produces correct results. 

a. Code verification is the process by which the structure, flow, and fidelity of the 
computational model are demonstrated to be correct with respect to the intended purpose (in 
accordance with the specifications). Verification of the structure and/or flow is by code tracing, 
unit testing, i.e., running the M&S through a series of low-level tests, and comparison of the 
coded model with the conceptual/mathematical models. Some, if not all, of the tests should be re- 
run any time the code is changed (a process known as regression testing) either to fix a software 
error or to add new functionality to ensure the changes do not introduce new errors. This topic 
also addresses the issue of code coverage, i.e., the percent of relevant logical branches within a 
code tested for proper numerical and logical execution. 

b. Calculation verification encompasses efforts to assess computational model 
correctness and numerical accuracy, independent of the physics being modeled. This requires 
consideration of numerous parameters associated with the numerical algorithms. These would 
include quantities such as solver tolerances and sampling intervals, with each having temporal 
characteristics, e.g., Runge-Kutta integration of differential equations, and spatial characteristics, 
e.g., type of mesh element and density of finite element meshes or density of ray bundles in 
thermal radiation models or in optical geometric ray-trace and physical diffraction models. 

Often, the choice of these values involves a trade between accuracy and run-time efficiency, and 
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it is important that the effect on numerical accuracy be quantified and propagated to the overall 
uncertainties for the M&S results. Clearly, the best possible means to assess the correctness and 
accuracy of the numerical solution is by comparison to a closed-form solution, which may be 
possible, especially for simple, low-order problems. For example, eigenvalues and eigenvectors 
for three- or four-degrees of freedom (DOF) spring-mass systems can be found algebraically, and 
these solutions can then be used to test any numerical methods (eigensolvers) developed to 
support dynamics analysis of large-order finite element models (FEMs). Such results provide a 
rough estimate of the lower bound on achievable numerical errors. This topic should also address 
the statistical basis of any probabilistic analysis or confidence-based assertions. 

An additional context for verification is not associated with the development of the M&S but 
rather with its current use and is commonly called input verification. This process and its 
artifacts are used to detect human errors, e.g., typographical errors or other incorrect/inadvertent 
interactions with the software. One common method is the echoing of all input data, including 
selections made by a mouse or other input devices, to a log file for comparison with the intended 
inputs. Confirmation that this and/or other methods were employed is advisable when reviewing 
M&S results. 

5.4.2.2 Examples 

Finite element codes such as the NASA structural analysis system (NASTRAN) compute 
conditioning and goodness numbers that provide implicit indications of numerical errors in the 
computational model. One example is the ratio of the diagonal terms of the stiffness matrix to the 
same diagonal term in the upper triangular factor computed during the static analysis procedure. 
A ratio higher than 1 0 8 could indicate possible model support problems or a high stiffness ratio 
in components at a grid point, and these areas should be investigated. A second example is a 
calculation of virtual work based on static displacements, which should be zero at force 
equilibrium. This number is known as epsilon, and the numerical results are generally deemed 
acceptable if <10' 5 ; otherwise, the model should be examined for support problems 
(inappropriate constraints, boundary conditions, or grounding in a structural FEM) or the 
improper use of infinitely rigid elements. (The term “support problems” refers to inappropriate 
constraints, boundary conditions, or grounding in a structural FEM.) 

Common examples in solution verification are ensuring the appropriate type of mesh element 
and mesh refinement for FEMs, such as those used to solve problems in structural mechanics and 
heat transfer. Some examples of the types of mesh elements are bars, plates, shells, or solids. The 
fundamental objective of mesh refinement is to increase the mesh density, i.e., reduce the spatial 
sampling between grid points, until no significant change is observed in the output quantities of 
interest. As is the case for analysis of thermal-elastic stability of the large cryogenic optical 
metering structure designed for the James Webb Space Telescope (JWST), the mesh density is a 
balance between predictive accuracy and computational efficiency and a non-optimal numerical 
error mesh density to achieve acceptable simulation run times. 

To demonstrate the JWST composite design had achieved Technology Readiness Level- (TRL-) 
6, experiments were performed on the backplane stability test article (BSTA), which is a small 
triad representative of a section of the full-scale (6.5-m (21.3-ft) diameter) structure that supports 
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the 18 beryllium mirror segments. (See figure 27, JWST Backplane Stability Test Article.) These 
experiments were used to validate the model of the composite structure. Before validation, a 
study was performed to determine an acceptable mesh density, balancing numerical error with 
run time. 



Figure 27 — JWST Backplane Stability Test Article 

( 1 /6 th full-scale cutout of the flight backplane) 

Based on extensive experience with precision composite structures, the structural analysts were 
confident that a highly refined mesh using 1-mm x 1-mm (0.04-in x 0.04-in) elements would 
have the desired accuracy for the intended use. Using a mesh this fine to model the entire BSTA 
would require on the order of 1 00 million DOF in the model and exceed the capabilities of the 
computer system, not only in the numerical sense but also with respect to visualization. For the 
full-scale structure, the situation would obviously be even worse. 

The strategy employed was to model the various piece-parts that comprised BSTA, as well as 
JWST’s primary mirror backplane. The meshes for these piece-parts were built using the 1-mm 
(0.04-in) mesh density (sometimes called the Gold Standard Mesh by the project team), and then 
a series of less refined models were systematically built, all the while being compared against the 
most refined mesh in terms of stiffness, distortion, and thermal forces. 

The goal was to stay within 1 0 percent of the highly refined mesh for all of these metrics. In the 
end, the so-called BSTA Standard Mesh met the accuracy requirement using 8-mm (0.3-in) 
elements. Applying this mesh density to the entire BSTA now required on the order of 1 .2 
million DOF, which was computationally tractable. 
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Visualizations of the two meshes are shown in figure 28, Nominal Analysis Mesh and Highly 
Refined Mesh, for two of the many piece-parts used to build BSTA and the JWST primary 
mirror backplane: the cap gusset and the spanner tube. 




1-mm (0.04-in) mesh 



Cap Gusset Spanner Tube 

Figure 28 — Nominal Analysis Mesh and Highly-Refined Mesh 


Figure 29, Failure Load Prediction versus Mesh Refinement, shows the typical results of a 
convergence study. The predicted static failure load for a component is plotted versus the 
refinement factor (lx, 8x, 16x, 32x) for four different models of varying mesh density and with 
the average failure load observed in testing. 



Figure 29 - Failure Load Prediction versus Mesh Refinement 


It is important that all possible uses of the model are considered when selecting the convergence 
metrics. In this case, the original intended use for the model was to predict thermal-elastic 
stability of the JWST backplane. Up to this point, dynamics (jitter) analysis had been done using 
a much simpler structural representation with only one- and two- dimensional (1-D and 2-D) 
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elements, e.g., bars and plates, and not the 3-D solid elements used for thermal distortion 
analysis. The project decided that maintaining two models would be problematic; hence, the 
solid-element model became the single source for both analyses. If stiffness had not been 
selected as one of the convergence metrics during the mesh refinement study, the numerical 
errors in the dynamics model would not have been quantified, and the BSTA Standard Mesh 
might have produced unacceptably large errors for jitter predictions. 

This example highlights a particular challenge with M&S built using COTS software. Because 
of its proprietary nature, COTS software typically does not provide evidence for verification 
Levels 1 and 2. However, by the fact that the software is widely used and accepted by the 
industry, it is assumed compliant with those levels. The model, which uses the NASTRAN 
software, was assessed at Level 4 for verification. The mesh refinement study (formal method) 
quantified the impact of mesh density on numerical error, and a density was chosen that satisfied 
the project-defined acceptance criteria (errors are small). 

5.4.2.3 Improving Credibility in Verification 

To successfully attain a given level of credibility for the verification factor, all lower level 
criteria have to be satisfied. Methods and suggestions to improve the credibility assessment for 
this factor are given in table 2, Achieving Verification CAS Factor Levels. 
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Tabl 

e 2 — Achieving Verification CAS Factor Levels 

Level 

Verification 

Evidence 

Needed to Achieve this Level 

4 

Numerical 
errors small for 
all important 
features 

Reliable error estimation 
methods are used to 
quantitatively assess numerical 
errors. These estimates show the 
errors are small from test suites, 
which exercise all important 
algorithms, all important 
features and capabilities, and all 
important couplings (physics, 
modules, etc.) of the full 
computational model. 

Determine the important features 
of the computation model using, 
for example, a sensitivity analysis 
approach. Apply formal numerical 
estimation, and demonstrate the 
errors associated with the 
important features satisfy 
requirements for the intended use, 
adjusting numerical parameters, 
e.g., sampling and tolerances, as 
required. 

3 

Formal 

numerical error 
estimation 

Some formal method is used to 
assess numerical errors 
associated with unit testing with 
significant coverage of the code. 

Quantify the effects of numerical 
errors, e.g., temporal sampling 
(time-domain simulation), spectral 
sampling (frequency response), 
and spatial sampling (finite 
element mesh, ray trace); 
tolerances used for iteration loops; 
finite machine precision. 

2 

Unit and 
regression 
testing of key 
features 

Favorable results from unit and 
regression testing of key 
features of the computational 
model 

Identify the key features of the 
computational model. Conduct 
unit tests to demonstrate correct 
behavior of the relevant parts of 
the code. Conduct regression tests 
when the code is updated (fixed or 
new functionality added) to 
demonstrate that no new errors 
were introduced as a result. 

1 

Conceptual and 
mathematical 
models verified 

Favorable evidence of 
verification for conceptual and 
mathematical models 

Show the M&S predictions agree 
with analytical solutions for simple 
systems. 

0 

Insufficient 

evidence 

Insufficient evidence 



5.4.3 Validation 

Validation is the process of determining the degree to which a model or a simulation is an 
accurate representation of the real world from the perspective of the intended uses of the M&S. 
This is sometimes referred to as anchoring the model and is based on comparisons between the 
simulation (computational) results and some referent Validation addresses uncertainties arising 
from both experimental and computational procedures. The term uncertainty is used in a general 
sense and can comprise a number of related terms, including the concept of error. 
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Validation is typically an iterative process involving multiple attempts at tuning model 
parameters, including those representing the system, controlling numerical accuracy, and in some 
cases the M&S assumptions and framework. Obtaining good correlation between predictions 
from the M&S and measurements from the RWS (or independent predictions in some cases) over 
the widest range of parameter space, initial conditions, boundary conditions, and modes of 
operation is desirable to maximize confidence. However, this is not always possible or 
affordable. Requirement 4.1.3a of NASA-STD-7009 identifies the conditions the 
program/project has to satisfy to achieve a favorable comparison between the M&S and the 
referent. 

A review of the validation process and results should address the following questions: 

a. What was the referent? 

b. What are the significant similarities and differences with respect to the RWS? 

c. Which uncertainties in the simulation and referent, e.g., numerical error, input data 
variability, measurement error, were considered when comparing the simulation output to the 
referent? 

d. What model and/or input data calibration (tuning, adjustment) was performed so that 
agreement between the referent and the predictions met the requirements for the intended use of 
the model? Was this justified? 

Note that calibration can be difficult for complex simulations, e.g., those for flight. 
There could be hundreds of changes needed to tune the model to match the RWS. 
Conversely, one change could make a good match for one scenario but could cause an 
issue for other scenarios. It is important to ensure the model not be overtuned so as to 
unnecessarily narrow the domain of validation. 

When reviewing the validation activities for a given M&S, it is important to identify known 
differences between the referent and the RWS. A referent may be the RWS to which the analysis 
is directed, or it could be a similar or analogous system, whereby the closeness of the referent to 
the RWS becomes pertinent. The Worksheet Results and Comments columns (figure 30, 
Worksheet Items: Validation) could be used to note the most significant differences and at least a 
qualitative assessment as to the effects these differences may have upon predictions for the 
RWS. When subsequently reviewing the uncertainty quantification methods and results, the 
M&S practitioner should account quantitatively for each estimate of uncertainty introduced by 
these differences and note what was done for model calibration, and what parameters were 
adjusted and by how much from their nominal values. 


Item | ^ | Result | Comments 

M&S Credibility Assessment (Development - Usage (Analysis) - Process] 

Validation 

How weO did test predictions using the 
M&S match referent data? 




How ebse is the referent to the real-world 
system, including its environment? 





Figure 30 — Worksheet Item: Validation 
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5.4.3. 1 Explanations 

As previously stated, validation involves comparisons between results of the computational 
model and a referent. Generally, the referent is experimental data and, in ideal circumstances, 
these data come from the real system being modeled or analyzed. However, the RWS is not 
always forthcoming, particularly in the development of novel systems. In such cases, analysis 
models have to obtain data from some analogous system for validation purposes. Two aspects of 
a validation referent are important and notionally depicted in figure 3 1 , Referent Similarity. 
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Figure 31 — Referent Similarity 


The bottom axis of figure 3 1 is used to indicate the quality of the referent system. As one moves 
from left to right in this depiction, the quality of the system data from the referent improves 
relative to the target system. If no equal system exists and the validation data are from similar 
systems, the quality of similarity is important to consider, and caution is warranted. In some 
cases, another M&S may be used as a referent for validation; in such cases, it is necessary to 
consider the similarity of the system being modeled with the referents used to validate the other 
M&S. 


The use of another M&S as the referent is a debatable but still common practice. For many 
programs/projects, the opportunity to validate the M&S through experimental measurements 
comes only after flight hardware is built and tested. Accordingly, throughout the earlier lifecycle 
phases, the choices for validation are either by review (sometimes called face validation) or by 
independent M&S. It is desirable for independent M&S to make use of different assumptions, 
methods, and software. Getting the same or similar results by independent means boosts 
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confidence in the original results by removing some of the epistemic uncertainty. Another 
situation where model-model validation is common involves the use of surrogate models 
(sometimes called metamodels). Surrogate models are typically created by exercising a high- 
fidelity model repeatedly, varying the inputs, and collecting the outputs to form response 
surfaces. The surrogate models, once validated, can then be used in place of the more complex 
model to achieve run-time efficiency. 

As the similarity of the referent approaches that of the target RWS, analysis credibility 
correspondingly improves. However, the environment of the referent system is equally 
important. If the referent system is sufficiently similar to the target system but operates in a 
different environment, then judgment of the referent data suitability is important. The real-world 
environment analogy is indicated along the vertical axis in figure 3 1 , showing improvement in 
referent quality while moving up the axis. Both axes depict a spectrum of possibilities, with full 
credibility achieved when an exactly matching referent system resides in the exact environment 
of the operational target system. 

It is important to note that throughout a program/project lifecycle the intended use of a given 
M&S can change. In the most general case, the M&S is created to support the design of the RWS 
to understand how well the design meets mission requirements. Then, the M&S evolves to 
analyze how the system design meets its specifications, eventually showing the system can safely 
perform its mission. Throughout this process, the fidelity of the M&S will likely improve, along 
with the quality of the input and possibly referent data, requiring the reiteration of validation. 

A typical process of referent data evolution might be: 

• Similar historical system. 

• Prototype, testbed, or alternative M&S for the new design. 

• The new design but not in the real environment. 

• The RWS in its real environment. 

In such a case, as the quality of the referent improves, better assessments of M&S validity are 
possible. However, the adequacy of M&S validation must still be reconsidered with respect to 
the lifecycle maturity of the RWS and the criticality of the decisions influenced by analyses with 
the M&S. 

The evolution in both the fidelity of the M&S and the quality of the referent used for validation 
should be considered, as these factors link to verification, uncertainty quantification, sensitivity 
analysis, and ultimately to predictive accuracy. In general, an M&S with less fidelity is a more 
cost-effective tool for exhaustively sweeping the parameter space in support of uncertainty and 
sensitivity analyses; however, it is typically only validated using lower quality referents. Yet, an 
M&S with higher fidelity, ostensibly validated using a higher quality referent, is typically the 
tool used for critical analyses. The combination of uncertainties from different M&S requires 
careful thought. Furthermore, lower quality referents, e.g., laboratory testbeds, may allow for a 
wider variety in range of experimental measurements than is possible with the RWS. 
Accordingly, the validation domain for the higher fidelity M&S may be smaller than that for the 
lower fidelity M&S; therefore, if the higher fidelity M&S is used for critical analyses, then there 
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is a trade between greater validity and greater risk of operating outside the more narrow 
validation bounds. 

5.4.3.2 Examples 

The first of three examples involves a gravitational model and illustrates the use of a second 
model as the referent. Specifically, it involves the approach developed to validate the Gottlieb 
spherical harmonic gravity acceleration and torque algorithms as implemented in the Johnson 
Space Center Engineering Orbital Dynamics (JEOD) simulation software. 

To model the orbital motion of a spacecraft realistically, the acceleration caused by gravity has to 
be computed accurately. Gravitational torque acting on the spacecraft has to be accurately 
computed for high-precision attitude modeling. The gravitational potential of a large, massive 
body, i.e., planet, is commonly modeled as a spherical harmonic series from which acceleration 
and torque can be computed. Most acceleration and torque algorithms are complex and involve 
mathematical recursions to compute high-order Legendre and trigonometric functions efficiently. 
Validation techniques exist for simple spherical and oblate planet models. A technique for 
validating general, higher order gravity algorithms was required. The technique described below 
validates gravity algorithms, not the specific coefficients of any gravitational body, and is 
therefore not limited to Earth models. 

A fictitious system of point masses was developed to represent a large gravitational body. For 
familiarity, this point mass planet was scaled to approximate the mass of Earth. Low-degree 
normalized spherical harmonic gravity coefficients were computed to represent the total 
gravitational potential of the point mass planet. The point masses were configured such that all 
gravity coefficients above degree two were non-zero and that coefficients of degree five (and 
higher) were at least several orders of magnitude smaller than the approximate limit of double- 
precision floating point arithmetic (15 significant figures). The assumption was that algorithm 
recursions that worked correctly to degree and order four also worked correctly for higher 
degrees and orders. However, to mitigate errors caused by truncation of the infinite spherical 
harmonic series, gravity coefficients through degree five were included in the model. The 
coefficients were used as data for the JEOD algorithms to compute acceleration and torque at 
various test locations external to the point mass system, including points over the north and south 
poles where mathematical singularities could present numerical problems. Simultaneously, the 
acceleration and torque vectors related to each point mass were computed directly from basic 
gravity principles and summed to give the total acceleration and torque acting at each test 
location. The total acceleration and torque were compared to those quantities computed using the 
JEOD algorithms for validation purposes. 

Level 2 for Validation was assessed, as M&S results compare favorably for unit problems at 
validation points by comparison of M&S results to an acceptable referent, which in this case are 
higher fidelity M&S results. The predictions from lower order harmonic series representation of 
a gravity model were compared to predictions from a higher order representation and found to be 
acceptably close for the intended use. 
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The second example, also from the JWST project, illustrates a common situation wherein 
validation leads directly to calibration or tuning of the model. Ideally, one or more experiments 
are conducted, with predictions made for each. When the results are compared, uncertainties in 
the predictions should account for all known unknowns, including numerical errors, i.e., 
quantified via the verification process, and model parameter variability. Many times, however, 
only the latter is considered and only implicitly through the use of an automated process in which 
model parameters are randomly or systematically adjusted, post-test, until some best match to the 
measurements is produced. 


Figure 32, Mesh for Prototype of Primary Mirror Segment Assembly, shows the mesh for a 
prototype of a Primary Mirror Segment Assembly (PMSA). 



Figure 32 — Mesh for Prototype of Primary Mirror Segment Assembly 


To validate the PMSA model for several different dynamics analyses, a modal survey test was 
conducted. The PMSA was suspended by bungees to achieve free- free boundary conditions and 
was instrumented with 49 accelerometers at 25 locations (12 triaxial and 13 uniaxial). Data were 
acquired using impact excitation at three locations on the mirror. The frequency content of the 
data was recorded to 2000 Hz with 14 modes extracted. 

The PMSA FEM was correlated using the following iterative procedure: 

a. Modal sensitivities were found for the design variables, i.e., model properties, using 
NASTRAN. 

b. The NASTRAN output, i.e., modal sensitivities and mass matrix, was input into the 
model correlation software Attune™, which runs in MATLAB®. Attune™ examines the design 
space to find a minimum correlation root mean square (RMS) error, including the frequency and 
modal orthogonality errors. The design variables are revised with the best correlation. 
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c. NASTRAN was run iteratively with the revised design variables. Then, Attune™ was 
used to find a new best correlation around the revised design variables. The design variables 
were revised and the process repeated until the correlation could no longer be improved. 

Correlation was based on minimizing the difference in the predicted versus measured frequencies 
for the important resonant modes of this structure. Before correlation, the differences for the key 
modes ranged from 4 percent to 9 percent; however, post-tuning the largest difference was less 
than 3 percent, which met the goal of 5 percent set by project staff and documented in JWST- 
REF-002290, James Webb Space Telescope Math Models Guidelines Document, Rev. C. Key 
resonant modes are those associated with DOF for the mirror assembly that result in significant 
impact on optical performance of the telescope. For example, rotations about two orthogonal 
axes in the plane of the mirror surface (tip and tilt) shift the position of the image on the focal 
plane. Compared to an ideal model validation process, neither the numerical errors in the 
predictions nor the measurement errors were accounted for in this process. Furthermore, after 
tuning, the modal test could have been repeated with some combination of different force input 
levels, impact locations, and measurement locations. If predictions using the calibrated FEM 
matched all the new measurements to within 5 percent, then the validation goals would be 
satisfied. If not, this would be a good indication the test failed to control temperature and the 
temperature-dependent material properties were not accounted for in the predictions, i.e., the 
model form was not correct. 

Level 3 for Validation was assessed, as M&S results compare favorably for problems of interest 
at validation points by comparison of M&S results to an acceptable referent, which are 
experimental measurements on problems of interest. This model does not achieve a higher score, 
even though these are the flight mirrors, because they are tested in a non-flight-like environment, 
i.e., at room temperature, under gravity load, and with non-representative boundary conditions. 

As a final example, accurate prediction of the power available from the solar arrays on the 
International Space Station (ISS) requires modeling of the location and amount of shadowing on 
the arrays. Analysis tools are available to predict array shadowing and its impact on the solar 
array current; these tools include several key assumptions, such as lower fidelity geometry 
models of ISS, minimal Sun subtense angle effects, and minimal reflected energy from adjacent 
hardware. With these differences between the model and the RWS, the model's results were 
compared with on-orbit flight video stills and flight telemetry, showing the model produces a 
good representation of the RWS (figure 33, ISS Power Prediction). 
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Figure 33 — ISS Power Prediction 


Level 4 for Validation was assessed, as M&S results compare favorably for the RWS at 
validation points by comparison of M&S results to an acceptable referent, which are 
measurements on the RWS. The model predictions are a good match to measurements from ISS 
in operation. 

S.4.3.3 Improving Credibility in Validation 

To successfully attain a given credibility level for the validation factor, all lower level criteria 
also have to be satisfied. Methods and suggestions to improve this credibility assessment are 
given in table 3, Achieving Validation CAS Factor Levels. 
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Table 3 — Achieving Validation CAS Factor Levels 


Level 

Validation 

Evidence 

Needed to Achieve this Level 

4 

Results agree 
with real-world 
data. 

M&S results compare favorably 
for the RWS at validation points 
by comparison of M&S results 
to an acceptable referent, which 
is measurements on the RWS. 

Compare M&S predictions to 
the RWS. 

3 

Results agree 
with 

experimental 
data for 
problems of 
interest. 

M&S results compare favorably 
for problems of interest at 
validation points by comparison 
of M&S results to an acceptable 
referent, which is an 
experimental measurement on 
problems of interest. 

Compare M&S predictions to 
experimental data from systems 
or problems more complex than 
unit problems and reasonably 
similar to the RWS. 

2 

Results agree 
with 

experimental 
data or other 
M&S on unit 
problems. 

M&S results compare favorably 
for unit problems at validation 
points by comparison of M&S 
results to an acceptable referent, 
which is either an experimental 
measurement or a higher fidelity 
M&S result. 

Compare M&S predictions with 
either experimental 
measurements for unit problems 
or predictions from another 
higher fidelity M&S, i.e., one 
with a validation factor score of 
2 or higher. Predictions have to 
meet program-/project-specified 
requirements. 

1 

Conceptual and 
mathematical 
models agree 
with simple 
referents. 

M&S conceptual and 
mathematical models compare 
favorably with general problem 
and textbook referents. 

The M&S specification agrees 
with the observed or assumed 
system behaviors. The M&S 
passes a set of necessary and 
sufficient sanity tests. 

0 

Insufficient 

evidence 

Insufficient evidence 



5.4.4 Input Pedigree 

The input pedigree factor strives to address the adequacy and/or quality of the inputs to the 
model, including their completeness, breadth, and accuracy for use in a particular simulation, and 
the eventual analysis recommendations. Models are generally considered as encapsulations of 
certain system characteristics (figure 34, General Model Diagram) to which a set of data is 
applied for a specific analysis. The input to a model broadly refers to the data used to obtain 
simulation and analysis results. The input does not address the model mathematics or structure, 
the processing of information within the model, or statements of uncertainty accompanying the 
results. The data can, however, include specific modifying parameters, with or without 
uncertainty, to the model or be used to set up and initialize the model. 


APPROVED FOR PUBLIC RELEASE — DISTRIBUTION IS UNLIMITED 


N AS A-HDBK-7009— MONTH DD, 2013 


Input 



Model 



Output 


Figure 34 — General Model Diagram 


Even an imperfect input can be used in a critical analysis but only if the associated uncertainty is 
identified. The central idea is to communicate clearly the credibility of the input used in the 
analysis based on various attributes of the data used. 


The following factors should be considered for each input to an M&S: 


a. Source of the data. 

(1) SME. 

(2) Document. 

(3) Database. 

b. Quality of the source. 

(1) Notional. 

(2) Informed. 

(3) Specified. 

(4) Derived. 

(5) Measured. 

(6) Similarity of analogous data source. 

c. Diversity of the data source; greater is often, but not always, better. 

(1) Single values, e.g., a minimum, maximum, or average from a particular source. 

(2) A set of historical values for this input from a number of sources. 

(3) Single versus multiple instances. 

d. Quantity of the source data. 

( 1 ) A single value. 

(2) A set of values. 

e. Form of the input used. 

(1) Deterministic. 

(2) Deterministic with spread. 

(3) Probability distribution or stochastic data. 
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For this Worksheet item (figure 35, Worksheet Item: Input Pedigree), the Result column should 
include the value of the credibility assessment. Other things to include are the following: 

• Total number of inputs to the model: percentage of inputs pedigreed. 

• The overall assessment of the quality and level of authority for the model input. 


|Tfk^ |Comments 

M&S Credibility Assessment | Development - Usage (Analysis) - Process| 


What is the quality of the Input Data? 




Input Pedigree 

How authoritative is the Input Data for this 
analysis? 




Figure 35 — Wor 

ksheet Item: Input Pedigree 


The Comments column can contain information on the qualifications for the input assessment. 
Further notes with regard to the high-level questions for this item and references to supporting 
documentation are also appropriate. 

5.4.4.1 Explanations 

a. Source of the Data. 

The goal for the source data used in any analysis is that it originates from an authoritative source, 
which could be an SME, a credible document, e.g., project documents, journal articles, test or 
operational results, spreadsheet, database, or another model. 

b. Quality of Data Source. 

The input to an M&S may have a variety of quality characteristics: 

(1) Notional - an uninformed estimate. 

(2) Informed - an educated or experienced estimate (minimum, most likely, or 
maximum). 

(3) Specified - from system requirements. 

(4) Derived - from knowledge or calculation from the general physical characteristics 
of the system (a value or expression from given or known set of data). 

(5) Measured - from direct knowledge (empirical readings) or calculation from the 
actual RWS. 

Understanding the data quality is critically important to the credibility of an analysis and spans 
the full spectrum from low (notional) to high (officially accepted operational or test data). The 
most authoritative sources are officially designated and documented, while less authoritative 
sources are not quite so formal. Less formal sources are not necessarily inferior; the intent of this 
qualification of the data source is to understand clearly where the data originates and whether it 
is a good source. 
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Test data can be superior to historical or quality record data but should be used cautiously. Test 
data obtained from a design of experiments generally make it possible to determine means and 
spreads accurately, while data with confusing changes in inputs and multiple outliers can make it 
difficult or impossible to perform rigorous data analysis. 

Even data from the best source may not have the highest quality, depending on factors such as 
the lifecycle phase of the RWS and the availability of historical and/or analogous data. Early in a 
project’s lifecycle, notional data are sometimes used for initial analyses. Whenever notional data 
are used, these data should be clearly noted. (This may also be noted in the M&S Results & 
Caveats item discussed in section 5.3.3 of this Handbook). The best case is for analysis 
accomplished on an RWS in operation for an extended time with plenty of officially documented 
data. If data are obtained from an analogous RWS, then the level of data similarity should be 
documented. (See section 5.4.3. 1 of this Handbook concerning the assessment of validation 
referents.) 

If the data are obtained from another model or analysis, the data credibility is tied directly to the 
credibility of the model or analysis from which the data were obtained. Appendix B.3.2 of 
NASA-STD-7009 discusses this dependency on data obtained from other models for input. In 
such cases, the input pedigree credibility level is limited to the credibility level of the model 
from which the data are obtained. 

c. Diversity and Quantity of Data Source 

The basic idea of diversity of source data is that data are increasingly and statistically more 
acceptable coming from more than one instance, item, and/or test. Information obtained from an 
SME may be simply a single value for a given parameter in a model, e.g., a minimum, an 
average, or a maximum, or a set of potential values. It is better if the source is empirical 
operational or test data. So, even if M&S input data are single (deterministic) values, it is better 
if that value is derived (calculated) from a set of data than from only one value. Additionally, if 
the data set from which the input is derived includes data from a variety of real world instances, 
then the resulting input will be more representative of the population. 

As an example, if the desired input to a model is the processing time for a Space Shuttle Orbiter 
in the Orbiter Processing Facility, then the input will be more representative of the population if 
data are obtained from multiple orbiters and various mission flows, i.e., process iterations. 

The more supporting data for a specific model input, the higher the quality of that particular 
input. Statistically, an average obtained from a set of 50 data points is much better than an 
average obtained from 10 data points. The same can be said of statistically determined 
probability distributions: the more data the better the resulting pdf discussed in section 5.4.4. 1 .d 
below. This aspect of the quantity of data directly relates to the upcoming topic of uncertainty, 
with smaller data sets having statistically larger uncertainty than larger data sets. 

Small sample sizes, particularly in historical data, give relatively inaccurate estimates of the true 
mean and typically underestimate true variability. For example, the more you drive your car, the 
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more likely you are to drive in all types of conditions; if you only measured drive time on a few 
sunny days, the effect of rain is missing. 

d. Form of Input Used 

As implied above, the input used in an analysis can take many forms, from textual to logical to 
numerical or mathematical. A deterministic (single- valued) input may be obtained directly or 
derived from a set of source data. If derived, the method of derivation should be made known. 
The value of a parameter used in an analysis may be obtained in a variety of ways. 

A more interesting and complete analysis may be obtained by using a span of possible parameter 
values in a Monte Carlo run of an M&S. For example, a model may be run with the values of 
certain parameters stepped through increments from the possible minimum to maximum values 
or using parameter values randomly selected within one or two standard deviations of the mean. 

An even better analysis is accomplished using probabilistic parameter values. If a set of data is 
available for a given parameter, statistical analysis of the data may produce a pdf that accurately 
represents the original data set but in a more general way. Stochastic data, or data representing 
how a process varies over time, are another probabilistic source. Such statistical functions are 
then used for the parameters) in Monte Carlo-type runs of the M&S by drawing random variates 
from the defined probability distribution. Probabilistic and stochastic analyses are more complex, 
requiring specific statistical methods for analyzing the outputs of multiple model runs. 
Beneficially, however, the results also include a statistically calculated uncertainty. 

Models typically use multiple inputs with a variety of pedigrees. Ideally, the effect of all of the 
inputs is to be considered when determining the overall input pedigree for a given M&S-based 
analysis. As a matter of pragmatism, a rigorous assessment as to the most influential inputs to an 
M&S is helpful in reducing the effort in this task. 

5.4.4.2 Examples 

For a system process analysis, one example of an input is the processing time at one location. 
Several choices are possible for this input. This is obtained from a data set of 200 observations, 
as described below: 

a. Average: 2.00 days. 

b. Uniform distribution of the range of possible values: [1.87 : 2.37] days. 

c. Triangular distribution using minimum, mean, maximum: [1 .87 : 2.00 : 2.37] days. 

d. Statistically fit probability density function: 

1.870625 + GAMM(1., 2.607787)/GAMM(l., 20.666568). 
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5.4.4.3 Improving Credibility in Input Pedigree 

Determining the credibility level for the input to an M&S is interdependent on the factors 
discussed above. These details of input pedigree are to be considered in determining the overall 
input pedigree credibility level. 

Table 4, Input Pedigree CAS Achievement, is to be read from the bottom up (like the credibility 
assessment), with the general idea that improvement is achieved when ascending the table. Note 
that these sub-factors for input pedigree are not strictly ordered and should be considered as part 
of the discussion in the overall assessment of input pedigree. 


Table 4 — Input Pedigree CAS Achievement 


Source 1 


Diversity/Quantity 3 

Form of Input 4 

RWS 

Official 



Another 

Model/Analysis 

Analogous 


Stochastic (pdf) or Empirical 
Function 


Historical 

Variety of Process Iterations 

Average with Spread 

SME 

Unofficial 

Variety of Instances 

Range of Values 

None 

Notional 

Amount of Data 

Deterministic 


Notes: 

1 . Source: The data obtained from an analogous real-world source may be better than that 


obtained from another model or analysis; however, the reverse can also be true. 

2. Quality: The data quality from an analogous source may be as good as data quality from the 
historical system. 

3. Diversity/Quantity: Having data from a variety of instances, e.g., Orbiter tail numbers, may 
be as good as having data from one instance over many process flows. 

4. Form of Input: Form, correct units, and appropriateness to scenario. 

Table 5, Achieving Input Pedigree CAS Factor Levels, provides guidance for improving the 
input pedigree. 
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Table 5 — Achieving Input Pedigree CAS Factor Levels 


Levels 

Input 

Pedigree 

Evidence 

How this Level is Achieved 

4 

Input data 
agree with 
real-world 
data. 

The input data compare favorably 
with measured data from the RWS, 
or the input data came from M&S 
with a summary credibility rating 
above 3.5. Uncertainty associated 
with the input data is known. 

All input data are of the most 
representative form and 
obtained from an adequate 
amount of diverse, historical 
data from formally documented 
and authoritative sources for 
the RWS. 

3 

Input data 
agree with 
experimental 
data for 
problems of 
interest. 

The input data compare favorably 
with acceptable measured referent 
data from problems of interest, or 
the input data came from M&S with 
a summary credibility rating above 
3.0. Uncertainty associated with the 
input data is known. 

Key input data are of a 
representative form and 
obtained from diverse, 
historical data traceable to 
formally documented and 
authoritative sources for the 
RWS or a close referent. 

2 

Input data are 
traceable to 
formal 

documentation 

The input data are traceable to 
formal documentation, or the input 
data came from M&S with a 
summary credibility rating above 
2.0. 

Input data are traceable to 
formal documentation. 
Notional data are documented. 

1 

Input data are 
traceable to 
informal 
documentation. 

The input data are traceable to 
informal documentation, or the 
input data came from M&S with a 
summary credibility rating above 
1.0. 

Input data are from informally 
documented sources, i.e., level 
of authority is not established, 
or an analogous system or 
M&S with a summary 
credibility rating above 1 .0. The 
attributes of the input data, e.g., 
forms, values, and units, are 
correct relative to the intended 
use. 

0 

Insufficient 

evidence 

Insufficient evidence 



5.4.5 CAS - Results Uncertainty 

The Results Uncertainty Worksheet item is complementary to the Uncertainty in the Estimate 
item in the M&S-based Analysis Results & Caveats section, which focuses on the overall 
uncertainty manifest in the results of an M&S-based analysis. (For further information, see 
sections 4.4 and 5.3.2 of this Handbook.) This section and item provide a more detailed and 
complete exposition on the topic, though by no means is it a complete coverage of the topic. 

The significance of uncertainty in the results depends on how the results are to be applied in a 
decision situation. The uncertainty in a given result may not matter in some situations, while in 
others it may imply that the nominal or best estimate result is suboptimal or even questionable. 
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In the latter case, if the decision stakes are high enough, it may be appropriate to invest in 
additional analysis or testing to reduce the uncertainty. Refer to the discussion of the decision 
robustness in section 1.5 of NASA/S P-20 10-576 and sections 6.4.2. 3 and 6. 8. 2. 8, including 
discussion of figure 6.4-6, in NASA/SP-2007-6105, NASA Systems Engineering Handbook, for 
additional information. 

The basic premise is that models are abstractions of actual or proposed RWSs, which necessarily 
induces some uncertainty in the model’s ability to replicate system behavior. Uncertainty 
characterization and quantification are difficult parts of understanding any system or model of a 
system. Deterministic analyses leave the uncertainties unaddressed and provide misleading, if 
not incorrect, results. Uncertainty presents itself in most aspects of modeling and, therefore, has 
its roots in system understanding, model building, input development, running models, and 
output analysis and thus spans the whole scope of the M&S process (figure 2). Figure 36, 
Locations for Contributing Factors Affecting Uncertainty, is an extension of figure 34 with 
general areas of uncertainty identified. 


Model 

(with Uncertainty) 


Output Post- 
processing 


Figure 36 — Locations for Contributing Factors Affecting Uncertainty 


Uncertainty comes in many forms and may present itself in a variety of places relevant to the 
analysis, including the following: 


• System understanding: how well the system is known. 

• Model building: what is and is not included in the model. 

• Input: the amount of good, i.e., attributable or authoritative, data available and the form 
the data take. (See section 5.4.4 of this Handbook.) 

• Running the models: the setup and initialization parameters for running the model. Do 
they meet the breadth of analyses required? Are the simulation model scenarios 
accomplished with a well-considered design of experiments? Are the numerical errors 
sufficiently small? 

• Output analysis: does the form of the output portray the breadth of the results obtained? 

Uncertainties are often classified into two separate types: 


• Epistemic - a lack of knowledge of the quantities or processes identified with the system, 
i.e., subjective, reducible, and may be identified with model uncertainty. If the system 
could be studied more closely, it may be possible to reduce the magnitude of the 
uncertainty 
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• Aleatory - the inherent variation in the physical system, i.e., stochastic or irreducible. 
Systems have inherent differences in their characteristics, which may change on a day-to- 
day basis. 

There are many potential sources of uncertainty in a model, with typical sources listed in figure 
37, Sources of Model Uncertainty, (Oberkampf, et al., 2002). This figure was made from the 
perspective of models based on partial differential equations; other types of models will not have 
some of these sources and yet have other sources of uncertainty. The A and E notations in figure 
37 refer to whether the uncertainty source is aleatory or epistemic. Furthermore, this figure 
distinguishes between epistemic uncertainties, aleatory uncertainties, and errors. For purposes of 
NASA-STD-7009, errors are considered uncertainties since they influence how well the model 
represents an RWS. 
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Figure 37 — Sources of Model Uncertainty 


The following information should be considered for the Results Uncertainty item. For large 
models, it may become necessary to focus on key sources of uncertainty. If this is the case, 
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additional caveats may be necessary, e.g., all sources of uncertainty are not documented or 
characterized. This may influence the risk associated with accepting the results of the M&S- 
based analysis. 

a. How were the uncertainties determined? 

b. How thoroughly were the uncertainties identified and evaluated? 

c. Are the sources documented? 

d. What are the sources of the uncertainties? 

(1) In the system? 

(2) Included in the model? 

(3) Excluded from the model that induces uncertainty? 

(4) In the data for, the parameters of, and the input to the model? 

(5) In the results/calculations of the M&S and analysis? 

e. What method(s) were used to quantify uncertainty, e.g., Monte Carlo, test data 
obtained using design-of-experiments principles, or Kriging-model-based survey data,, including 
how uncertainty propagates through the model to the results? 

f. Were the types of uncertainty documented? 

(1) Epistemic. 

(2) Aleatory. 

(3) Error. 

g. How well is the uncertainty known? 

h. What is the magnitude of the uncertainty?. 

i. Is there an Uncertainty Mitigation Plan?. 

The Result column for this item, shown in figure 38, Worksheet Item: Results Uncertainty, 
should include the value of the credibility assessment and whether all sources, locations, types, 
and magnitudes of uncertainty are listed. 


Item TTJS^ii 

(Comments 

M&S Credibility Assessment (Development - Usage (Analysis) ■ 

- Processl 




What methods are used to analyze the 





Results Uncertainty 

uncertainty in the results of this analysis 
(including sources and propagation)? 






Figure 38 — Worksheet Item: Results Uncertainty 


The Comments column can include qualifications of information in the Result column, along 
with items pertaining to the actions and plans to reduce the uncertainty. 
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5.4.5.1 Explanations 

This section discusses details associated with types of information needed to more fully 
understand uncertainty in M&S: 

a. Sources: Listing what is not known or not fully known in an M&S is a beginning. 
Each item can then be enhanced with some qualifying information. 

b. Location: Knowing where uncertainties are located in the RWS aids in understanding 
it and also in determining whether or not these uncertainties should be included in the model, 
e.g., if the magnitude of an uncertainty is small relative to other parameters in the system or 
inconsequential to the outcome, then it may not be needed. Knowing the architecture of the M&S 
and the locations of the uncertainties can help understand how uncertainty propagates through 
the model to the results. 

c. How well known: An analyst may know there is something not known about a part or 
parameter of the RWS but not know anything else. 

d. Magnitude: The magnitude of an uncertainty may be given in qualitative or 
quantitative form. If little is known about a particular system, then knowing a parameter may 
vary in a small or large way is useful. For example, knowing the clearance height of a high-value 
satellite processing facility door requires more than qualitative specification. 

e. Uncertainty Mitigation Plan: For critical parameters with uncertainty, it may be useful 
to develop a plan for reducing that uncertainty 

One method for tracking and qualifying the uncertainties in an M&S is by using a table similar to 
table 6, Sample Table for the Uncertainties of a Process. 


Table 6 — Sample Table for I 

he Uncertai 

inties of a Process 

Name 

Source 

Location 

Included 
in M&S? 

Type 

Well 

Known 

Magnitude 

Mitigation 

Plan 


























5.4.5.2 Examples 

a. Process Time Example 

For the second example in section 5.3. 1.2 of this Handbook (the timeline analysis for a single 
process), uncertainty analysis yields a broader understanding of the results presented (table 7, 
Example Table: For the Uncertainties of a Process). 
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Table 7 — Example Table: For the Uncertainties of a Process 


Name 

Source 

Location 

Included 
in M&S? 

Type 

Well 

Known 

Magnitude 

Mitigation 

Plan 

Prep Time 
Process 

Historical 

Data 

Launch 

Vehicle 

Stacking 

Facility 

Process 

Yes 

Aleatory 

Yes: 200 
data points 


No: 

uncertainty 

well 

known 


To represent the Prep Time Process (table 7) in the model, several types of data can be used: 

( 1 ) Detenninistic Representation: to represent the process time without uncertainty, 
one of three values are typically chosen: 

A. 2.00 days to represent the average case. 

B. 1 .87 days to represent the best (optimistic) case. 

C. 2.37 days to represent the worst (pessimistic) case. 

(2) Stochastic Representation: to represent the process time with uncertainty, a 
probability distribution may be chosen or statistically determined, based on 
available data: 

A. Uniform distribution - U(1 .87 : 2.37). 

If little else is known about the process time other than the extremes 
(minimum and maximum values), a uniform distribution allows random 
values to be generated during the simulation run from this mathematical 
model with equal probability. 

B. Triangular distribution - T(1.87 : 2.00 : 2.37). 

When the extremes and most likely value of the process time are known, a 
triangular distribution allows random values to be generated during the 
simulation run from this mathematical model with the highest probability 
of values coming closer to the most likely value. 

C. pdf - Pearson VI(E). 

When a set of values is available from the RWS process, a pdf may be 
statistically determined, from which random values are generated during 
the simulation run that statistically match the available data from the 
system. 
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i. Represented as 1 .870625 + GAMM(1 2.607787)/GAMM(l 
20.666568). 

ii. Graphically represented in figure 39, Example of a Right Skewed 
Distribution. 



Figure 39 — Example of a Right Skewed Distribution 

b. Box Example 

A simple example in evaluating uncertainty is the determination of the volume of a box from a 
picture (figure 40, 3-D Box Example). The volume is represented as a deterministic model. 
Assuming a box of dimensions width (W), depth (D), and height (H), the volume (V) is: 


V = W * D * H 



Figure 40 — 3-D Box Example 

The two dimensions, H and W, can be directly measured, e.g., with a ruler. Uncertainties in these 
two dimensions are related to variations in image replication, accuracy in the ruler, ability of the 
ruler user, process for measurement, e.g., if a measurement outside, inside, or in the middle of a 
line, errors, e.g., recording the wrong value, and other variations. The type of uncertainty from 
these various sources falls into two categories: aleatory and epistemic. An example of an 
aleatory uncertainty is the variation in image replication, since this is a stochastic process that 
does not change as additional information is gathered. Alternatively, an example of epistemic 
uncertainty is the recorded measurement, since this could change as additional measurements are 
taken, improved-accuracy methods are used, or the measurer becomes better with practice. 

The third dimension, D, requires inference, since this distance may be arbitrarily to make the 2-D 
picture look more realistic. The inference process could assume that D is the same as W and H 
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(if they are identical) on the presumption that the figure is a symmetrical box. Alternatively, the 
parameter D could simply be measured with the ruler and taken at face value, or a guess could be 
made based upon the values of W and H and the projected third dimension. This uncertainty 
from the degree of knowledge of D is classified as epistemic. 

These measures, the associated uncertainty, and other factors are included in table 8, Example 
Table: For the Uncertainties in the Analytical Volume of a Box. 


Table 8 — Example Table: For the Uncertainties in the Analytical Volume of a Box 


Name 

Source 

Location 

Included 
in M&S? 

Type 

Weil 

Known 

Magnitude 

Mitigation 

Plan 

Height 

Ruler 

measurement 


No 

Aleatory 

Yes 


No 

Width 

Ruler 

measurement 

Picture 

width 

No 

Aleatory 

Yes 


No 

Depth 

Ruler 

measurement 

and 

Picture 

depth 

No 

Aleatory 

Yes 

±1.6 mm 
(±1/16 in) 

No 

Unknown 
scale factor 


Somewhat 

Epistemic 

No 

Unknown 

Yes 


5.4.5.3 Improving Credibility in Results Uncertainty 

The amount of uncertainty analysis is dependent on the criticality of the situation, though the 
exact amount is not genetically detenninable. As with the other CAS factors, this is 
accomplished on a case-by-case basis. Ideas for improving the uncertainty assessment are shown 
in table 9, Achieving Results Uncertainty CAS Factor Levels. 
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Table 9 — Achieving Results Uncertainty CAS Factor Levels 


Levels 

Results 

Uncertainty 

Evidence 

How this Level is Achieved 

4 

Non- 

deterministic 
and numerical 
analysis 

Uncertainty estimates are 
quantitative and based upon 
nondeterministic and 
numerical analysis. 

Full probabilistic/stochastic 
analysis performed from 
model construction, to data 
input, running the M&S, and 
analysis of output, with a 
complete understanding of 
uncertainty propagation 
through the model. 
Uncertainties for all results are 
provided quantitatively. 

3 

Non- 

deterministic 

analysis 

Uncertainty estimates are 
quantitative and based upon 
nondeterministic analysis. 

Uncertainties for key results 
are provided quantitatively 
through non-deterministic 
analysis. 

2 

Deterministic 
analysis or 
expert opinion 

Uncertainty estimates are 
quantitative and based upon 
deterministic analysis or 
expert opinion. 

Uncertainties expressed 
quantitatively from SME 
opinions and/or as notional 
spreads of deterministic 
values. Propagation of the 
uncertainties into the results 
should be addressed. 

1 

Qualitative 

estimates 

Uncertainty estimates are 
qualitative. 

Sources of uncertainty 
identified and qualitatively 
addressed 

0 

Insufficient 

evidence 

Insufficient evidence 



5.4.6 Results Robustness 

Results Robustness is attributed to how thoroughly the sensitivities of the current M&S results 
are known, with some of these variables and parameters intrinsic to the RWS and others intrinsic 
to the M&S. Since the model is used to understand how changes in the various parameters 
impact the RWS, the sensitivities of the model should be similar to the sensitivities of the RWS. 
The robustness of the model results is one of the factors to assess the credibility of the analysis 
(Requirement 4.7.1 in NASA-STD-7009). The justification for the evaluation and any technical 
review of Results Robustness needs to be documented (Requirements 4.7.2 and 4.1.5 in 
NASA-STD-7009). 

Notes: 

( 1 ) NASA-STD-7009 defines sensitivity analysis but only references robustness in terms of 
sensitivity. This can lead to confusion about both terms, so some clarification is provided 
here. With respect to systems and models, sensitivity and robustness are opposites. If a 
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system is sensitive to relatively small changes in operating parameters or conditions, then 
it is not considered robust. On the other hand, if the system is found to be insensitive to 
relatively small changes in operating parameters or conditions, then the system is 
considered robust. Sensitivity analysis is the technique to better understand system 
robustness. , 

(2) The closeness of a model’s response to the system’s response should be part of the M&S 
validation effort. The Results Robustness CAS factor focuses on the degree to which 
sensitivity analyses were accomplished. If documentation is provided comparing the 
sensitivity of model results to the sensitivity of the RWS, then the requirement of 
NASA-STD-7009 is met. 

(3) Sensitivity analysis can also be used early in the RWS lifecycle, when limited validation 
data are available, to determine the boundaries for stable system performance. This is 
also useful when good referent data are not available. If system instability is indicated, 
then more attention is required to the affected portions of the system as it progresses in 
development (Kelton, et al., 2004). If system performance is adequately stable, i.e., 
insensitive to small changes in operating parameters, then margin may be available as the 
system design matures. 

This Worksheet item is shown in figure 41, Worksheet Item: Results Robustness. 


Item | /|kesull konm*n,s 


M&S Credibility Assessment | Development - 1 sage (Analysis) - Process! 


Results Robustness 

What are the significant sensitivities of the 
M&S results? 


CAS Value: 1.7 

Weighting Used? Yes 

Note significant sensitivities - 
% of parameters for which sensitivities were explored - 

How thoroughly are the sensitivities known? 



How much testing was performed to characterize the results sensitivity? 


Figure 41 — Worksheet Item: Results Robustness 


Additional considerations with respect to these key questions are: 

a. What are the significant sensitivities of the M&S results? 

(1) Which parameters, when varied, have the largest impacts on the results? 

(2) Do they match the sensitivities of the RWS? 

b. How thoroughly are the sensitivities known? 

(1) What percentage of parameters have had their sensitivities evaluated? 

(2) How much testing was performed to characterize the sensitivity fully? 

The overall assessment of this factor should be noted in the Results column. Significant 
sensitivities in model results as compared to the RWS should be noted in the Comments column, 
along with any statements concerning the quantity or percentage of model parameters to which 
the model results show sensitivity and lead to different decisions. 
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5.4.6.1 Explanations 

Results Robustness is concerned with how the results of an M&S-based analysis change as a 
result of changes in the parameters, variables, and conditions. For this factor, the CAS score is 
determined using the guidance provided in table 3 of NASA-STD-7009, along with the 
additional guidance provided in section 5.4.6 of this Handbook. Model and RWS sensitivities 
should be compared and documented. 

Note: Sensitivity can vary throughout a time-dependent model and, unless only the end state is 
the output metric of interest, these sensitivities over the course of the simulation should be 
identified. Additionally, sensitivities can be linear or non-linear. In some systems analyses, 
solved over large temporal and/or spatial extents, non-linear sensitivity analysis may be 
appropriate. 

For the situation in which an M&S and its sensitivities are validated and acceptable for use in 
analyzing an RWS and the analysis results show sensitivity to relatively small changes in 
operating parameters, the RWS should be operated close to those parameters. On the other hand, 
in the case of a novel RWS, i.e., where no validation data exist, and the M&S of the system 
indicates sensitivity to relatively small changes in key operational parameters, additional efforts, 
e.g., testing the RWS or data gathered from an analogous system, should be undertaken to better 
understand system behavior in the region of operation. 

There are cases where a robustness assessment or sensitivity analysis can provide valuable 
qualifying information to an M&S-based analysis. 

a. In the two cases listed below, M&S-based results are most likely acceptable: 

(1) The RWS is robust, and the M&S response is validated as similarly robust. 

(2) The RWS is sensitive (not robust), and the M&S response is similarly validated as 
sensitive. 

b. In the next two cases, caution is warranted and is a validation issue: 

(1) The RWS is robust, but the M&S response shows sensitivity to relatively small 
changes in variables or parameters. 

(2) The RWS is sensitive (not robust) to changes in operating parameters, but the 
M&S response indicates insensitivity to such changes. 

5.4.6.2 Examples 

Examples will be developed and provided in a later revision. 
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5.4.6.3 Improving Credibility in Results Robustness 

Table 10, Achieving Results Robustness CAS Factor Levels, provides guidance for improving 
the assessment of Results Robustness. 


Table 10— Achieving Results Robustness CAS Factor Levels 


Levels 

Results 

Robustness 

Evidence 

How this Level is Achieved 

4 

Sensitivity 
known for most 
parameters; key 
sensitivities 
identified 

Sensitivity of the M&S results 
for the RWS is quantitatively 
known for most of the variables 
and parameters, including all of 
the most sensitive variables and 
parameters. 

All key sensitivities that would 
drive model results are identified. 
Sensitivity of the results to most 
(>50%) of the model parameters 
has been quantified through 
sensitivity analysis. 

3 

Sensitivity 
known for 
many 
parameters 

Sensitivity of the M&S results 
for the RWS is quantitatively 
known for many variables and 
parameters. 

Sensitivity of the results to many 
(20-50%) of the model 
parameters has been quantified 
through sensitivity analysis. 

2 

Sensitivity 
known for a 
few parameters 

Sensitivity of the M&S results 
for the RWS is quantitatively 
known for a few variables and 
parameters. 

Sensitivity of the results to some 
(<20%) of the model parameters 
has been quantified through 
sensitivity analysis. 

1 

Qualitative 

estimates 

Sensitivity of M&S results for 
the RWS is estimated by analogy 
with the quantified sensitivity of 
similar problems of interest. 

Sensitivity of the M&S results is 
assumed by comparison to the 
quantified sensitivity of a similar 
problem. 

0 

Insufficient 

evidence 

Insufficient evidence 



5.4.7 Use History 

The Use History factor in the CAS describes the extent of prior favorable uses of the particular 
M&S in similar situations. Favorable means the M&S satisfied relevant acceptance criteria 
deemed sufficient by the program/project management in collaboration with the Technical 
Authority (NASA-STD-7009, Requirement 4. 1 .3(a)). The two dimensions to consider for a 
specific M&S result are the time of successful M&S use and the types of problems for which it 
was used. The central idea for Use History with regard to credibility is the longer a given M&S 
is used and the closer the historical use is to the current use, the more credible the results. While 
this is not a guarantee of good results, it is an indicator of the past successful trials of the M&S 
and, therefore, a point of discussion. 

This item (figure 42, Worksheet Item: Use History) includes a single general probing question, 
which can be further understood with the following details: 

• The time length of use for the M&S. 

• The number of analyses accomplished with the M&S. 
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• The similarity of the previous analyses with the current use. 

• When the model was most recently changed. 

• The favorable comparison, e.g., accuracy, of M&S results with the data. 


Item | •/ [Result 

M&S Credibility Assessment [Development - Usage (Analysis) - Process] 


Use Hist on 


How have the current M&S been 
previously used? 


Figure 42 — Worksheet Item: Use History 


These suggested details can give a sense of the historical use of the M&S (how long it has been 
used; how much it has been used; for what has it been used; how accurate analysis results have 
been). However, when an M&S is changed, it may also change its domain of intended use. 
Additional scrutiny may be required of an M&S that has recently changed to understand its 
intended use, V&V status, and relevancy for current use. 


The Results column should note the CAS factor assessment and list significant cases of prior use. 
In the Comments column, supporting details and documentation regarding those instances, along 
with the similarity of analyses performed, should be recorded. Major departures from primary 
uses of the M&S should be annotated, if relevant to the current analysis. 


5.4.7.1 Explanations 


When using COTS M&S software, insight into verification software testing can be limited. In 
some cases, source code, test suites, and data are available or may be made available with 
appropriate non-disclosure agreements. The goal is to obtain past-use information supporting the 
use of the COTS M&S software for an analysis. 

Exercise caution when using a particular M&S with a history of application to similar problems, 
as it is possible the current application uses a new version and runs on a different platform 
(central processing unit, operating system, compiler) or has added features, e.g., supports certain 
detailed physics previously unmodeled, used for the first time. 

It is recommended to examine carefully simple, qualitative arguments about the closeness of 
historical use of the M&S to the current use. 


5.4.7.2 Examples 

5.4.7.2.1 Example 1 : Choice of optical modeling, e.g., geometric ray trace and physical 
diffraction, code for the Laser Interferometer Space Antenna (LISA) mission. 

The following past uses of the Code V or Zemax modeling platforms argued in favor of 
developing a custom code: 
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a. For optical systems modeling, does the Use History of an existing model show that 
the issue of scale (wavelength versus system dimensions versus M&S precision) is adequate to 
the needs of the current analysis? 

b. For a system designed to operate at very short wavelengths or at longer wavelengths 
over large scales, which may require quad precision so numerical errors are insignificant, e.g., 
collector apertures sited on individual spacecraft flying in formation to realize a very long 
baseline interferometer), does the Use History of an M&S show that it supports the required level 
of quad precision? 

5.4.7.2.2 Example 2: Choice of thermal modeling, e.g., conduction, convection, and radiation 
heat exchange, code by the JWST mission. 

What are the comparative Use Histories of the previously used Thermal Synthesizer System 
(TSS) and Systems Improved Numerical Differencing Analyzer (SINDA) models compared to 
the alternate Thermal Desktop® and Thermal Model Generation (TMG) codes? What are the 
details requested in section 5.4.7 (above)? Detailed questions such as the following should be 
addressed when comparing historical and current usage: 

a. How long have these modeling codes been used and qualitatively for how many 
analyses: a few? many? 

b. Was the ray-trace solver used to compute radiation heat exchange factors? Was that 
similar to how the current application will use it? 

(1) Was past use of the scattering model similar to what is needed for the current 
analysis? 

(2) Were the types of surfaces analyzed previously using these models similar to the 
current system? 

(3) Are different coatings, materials, or fabrication processes used for the RWS such 
that the fine details of the surface invalidate the scattering model used? 

c. How well does this code integrate with other discipline codes, e.g., Mechanical 
Computer-Aided Design, structural FEM, and stray light, with which it must exchange 
information? Are there pitfalls? 

5.4.7.3 Improving Credibility in Use History 

To attain a given credibility level for the Use History factor successfully, all lower level criteria 
are to be satisfied. Table 1 1, Achieving Use History CAS Factor Levels, provides guidance for 
improving the assessment of this factor. 
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Table 11- — Achieving Use History CAS Factor Levels 


Level 

Use History 

Evidence 

How to Achieve Each Level 

4 

De facto standard 

De facto standard 

Use an M&S that is used 
extensively or exclusively within the 
relevant community of practice. 

3 

Previous 
predictions were 
later validated by 
mission data. 

Post-decision real world 
events have been 
represented accurately in 
results, e.g., validated by 
mission data. 

Provide evidence the M&S satisfies 
the criteria for Validation Level 4 
for one or more prior applications. 

2 

Used before for 
critical decisions 

Used previously to 
perform analysis upon 
which critical decisions 
have been made 

Provide evidence the M&S was 
used for critical decisions. Identify 
the mission and the instance(s) in 
which the M&S was used. 

1 

Passes simple tests 

Specific scenarios have 
been created to test 
application, or results 
compare favorably with 
outputs from other similar 
tools. 

Document the successful 
completion of development 
(including V&V) of this M&S for 
the revision used in the current 
analysis. 

0 

Insufficient 

evidence 

Insufficient evidence 



5.4.8 M&S Management 

The M&S Management factor provides supporting evidence of M&S credibility by describing 
the extent to which an M&S activity defines, follows, and documents a formalized planning and 
implementation process, which is similar to the Capability Maturity Model® Integration SM 
(CMMI sm ) 22 level developed by the Software Engineering Institute at the Carnegie Mellon 
University, including: 

• Work product management. 

• Process definition. 

• Process measurement. 

• Process control. 

• Process change. 

• Continuous improvement. 

• Configuration Management (CM). 

• Support and maintenance. 

This item, shown in figure 43, Worksheet Item: M&S Management, indicates a single general 
question about the administration of the M&S process. 


22 http://www.sei.cmu.edu/reports/02tr0 1 2.pdf . Retrieved April 23, 2013. 
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Item 

| / 1 Result 

IComments 

M&S Credibility Assessment (Development - Usage (Analysis) 

- Process | 


M&S Management 

What formal processes were used in the 
devebpment & use of this M&S? 





Figure 43 — Worksheet Item: M&S Management 


Additional key issues for assessing the M&S Management CAS sub-factor include the following: 

a. Was there a defined formal process for M&S development and use, i.e., was there a 
project plan? 

b. How CM was accomplished for the M&S and data? 

(1 ) Is there an M&S and data repository system? 

(2) What versions of M&S and data were used for the current analysis? 

c. If the analysis system is comprised of a set of coupled models, was there an ICD? 

The Results column can be used to document the assessed credibility level with notes, e.g., no 
evidence of formal M&S management process. The Comments column could be used to record 
additional details or identify reference documents. 

5.4.8.1 Explanations 

There are a variety of processes potentially associated with a given M&S. Each needs to be 
understood and the details probed to identify and assess aspects that may lead to credibility 
issues. These processes often are repeated over a program’ s/projecf s lifecycle, although they are 
likely to evolve to support the immediate objectives. 

It is insufficient to ask the question “Were formal processes defined and implemented?” Proof 
they were followed is essential to adding to the overall credibility of the M&S through 
supporting evidence of good planning and use of best practices and providing information for the 
reviewers or decision makers to examine the processes and identify potential areas of concern. 

5.4.8.2 Examples 

An example of a higher level process is the JWST Program shown in figure 44, JWST Integrated 
Modeling Cycle. Multiple cycles supported requirements development, trade studies, and 
preliminary and detailed design. This process involved the following progression from: 

a. Defining the system/problem. 

b. Defining the M&S approach. 

c. Developing, inspecting, and reviewing component-level models. 
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d. Assembling, inspecting, and reviewing system-level models. 

e. Executing simulations, and conducting analyses. 


Observatory Model 

Design Configuration Analysis 

7 Review V Review Review V 


Review 
Results & 
Prepare for 
Next Cycle 

j 


Figure 44 — JWST Integrated Modeling Cycle 

Nested within this process were multiple sub-processes. The development/inspection/review 
processes conducted at the component and system levels involved performing sets of sanity 
checks or verifications to ensure model workmanship, i.e., the models were built to specification, 
the acceptance criteria were met, or when not met, the impact was understood. The associated 
activities involved handoffs between models, with verifications repeated on both sides of the 
interface, similar to hardware integration activities. System model integration within individual 
disciplines or domains was accomplished using ICDs to ensure compatibility of coordinates, 
units, and geometries, including FEM meshes. Reviewers of such M&S processes should 
examine model interfaces, especially when component and system models are developed by 
different organizations or when models are translated between applications. For an integrated set 
of models used in analyses, examine how model/data interface control was formally established, 
e.g., in a Model-to-Model IDD/ICD. 

The JWST simulations and analyses involve a complex process, shown in figure 45, JWST 
Interdisciplinary M&S Workflow 23 , linking multiple M&S both in series and parallel. This 
ensemble of M&S is used to evaluate system-level performance under specified operational and 
environmental conditions, relative to a set of technical perfonnance metrics. As with 
discipline/domain model integration, this process is rife with interfaces between organizations 
and different software tools and should be examined closely. 



Determine 

System Component 

Configurations ^ Model 

to be Reviews 

Analyzed ___ 


Integrate & 
Debug, 
Model 
Check 
Observatory 
Models 


Mission CDR Analysis Cycle 



23 http://proceedings. spiedigitallibrary.org/proceeding. aspx?articleid= 1200449 . Accessed April 24, 2013. 
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Figure 45 — JWST Interdisciplinary M&S Workflow 


It is important to balance the depth and breadth of the various aspects of M&S management 
against the scope of the M&S effort. It is possible to impose M&S management to the detriment 
of M&S quality. In a JWST case, with the M&S activity comprising numerous programmatic 
and technical interfaces, significant investment in infrastructure and process was essential. On 
simpler or less critical M&S efforts, complex controls may not be warranted. 

The M&S management factor for credibility (like V&V) has to meet all lower level criteria to 
attain a particular assessment level. The JWST M&S management factor was assessed at Level 
3, as follows: 


• Meets Level 1 : The roles and responsibilities for the M&S team were defined by JWST 
Project Management (NASA), Mission Systems Engineering (NASA), and Observatory 
Systems Engineering (Northrop Grumman). 

• Meets Level 2: The M&S was developed, operated, and configuration controlled 
according to formal procedures defined in two documents: JWST-REF-002290 and 
JSWT-PLAN-006165, James Webb Space Telescope (JWST) System Modeling and 
Analysis and JWST Models Validation, Verification and Calibration Plan. 

• Meets Level 3: Independent review performed by NASA Goddard Space Flight Center 
(GFSC). The M&S team periodically demonstrated repeatability of the M&S results. 

There are many possible choices for M&S management strategies and their implementations. It 
is recommended the party responsible for M&S management (sections 1.1 and 5.1 of this 
Handbook) document the key elements and status of the chosen strategy. One example for 
documenting the M&S management effort is shown in table 12, Elements of M&S Management. 
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Suitable M&S management plans should be coordinated with the software management plans 
required by NPR 7150.2, NASA Software Engineering Requirements. 


Table 12 — Elements of M&S Management 


Sample Elements of M&S 
Management 

Example Disposition/Status Notes 

Project Plan 

Project plan was documented and is current with the M&S. 

Production Repository System 

Repository system is used for the M&S development products 
and documentation, test scenarios and results, and analysis 
products. 

Version Control 

Model versions are controlled. 

Data (relevant, current, 
authoritative) 

Data are traceable to Preliminary Design Review baseline 
configuration Rev X (dated dd/mm/yyyy) obtained from 
(source, individual, or program/project design data 
repository). 

Interfaces 

Model/data interfaces are documented and controlled via 
XX-ICD-xxxx dated (dd/mm/yyyy). 


5.4.8.3 Improving Credibility in M&S Management 

Table 13, Achieving M&S Management CAS Factor Levels, provides guidance for improving 
the assessment of M&S Management. 


Table 13 — Achieving M&S Management CAS Factor Levels 


Level 

IHEHI 

Evidence 

How to Achieve Each Level 

4 

Continuing Process 
Improvement 

The M&S effort is using 
measurements on M&S 
processes to improve the 
repeatability of the M&S 
results. 

Defined M&S metrics and 
processes are used to correct the 
models, simulations, and overall 
M&S processes and measure 
repeatability of M&S results. 

3 

Predictable Process 

The M&S effort is 
measuring repeatability 
of the M&S results 
generated by the M&S 
processes. 

Define the metrics and processes 
needed to measure repeatability of 
the M&S results. Document 
repeatability measurements 
outcomes. 

2 

Established Process 

The M&S effort has 
established a documented 
process for M&S 
development and 
operations. 

Formally document the processes 
for M&S development, operations, 
and compliance. 

1 

Managed Process 

The M&S roles and 
responsibilities have been 
defined. 

Identify M&S process management 
roles and responsible parties. 

The process for managing M&S 
products is informally documented. 

0 

Insufficient 

evidence 

Insufficient evidence 
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5.4.9 People Qualifications 

The People Qualification factor is used to assess and understand the qualifications, e.g., 
education, training, and experience, of the personnel involved in the M&S activities. The 
inclusion of this factor is an important aspect to the overall consideration of results credibility. 
These qualifications should be related to the specific M&S and its underlying discipline, e.g., 
science, math, and engineering. Additional details are discussed in Appendix B.3.3.3 of 
NASA-STD-7009. 

Note The desired level for the assessed value and the contributing factors should be accepted by 
the project/program management in association with the Technical Authority, as appropriate for 
the current state of the system, M&S development, and the analysis/decision. 

Figure 46, Worksheet Item: People Qualifications, indicates a single general question about the 
qualifications of the M&S practitioners. 



Item 

| / 1 Result 

Comments 

M&S Credibility Assessment (Development - Usage (Analysis) ■ 

■ Processl 


People Qiiulificiil ions 

What are the qualifications & experience of 
the people developing, testing, & using this 
M&S? 





Figure 46 — Worksheet Item: People Qualifications 


Additional points to consider are: 

• The qualifications and experiences of the developer, operator, and analyst. 

• Who performed the credibility assessment? 

• Evidence supporting the People Qualifications assessment. 

The overall assessed level of the qualifications for the people involved with the M&S can be 
annotated in the Results column (table 14, People Qualifications and Experience Example 
Table). The Comments column may include any additional information or notes needed to satisfy 
or better understand the People Qualifications information and/or any technical review results for 
this item. 


Table 14 — People Qualifications Example Table 



M&S Developer 

M&S Operator 

M&S Analyst 

Educational Background 




Experience in M&S 
Discipline 




Training for the Specific 
M&S 




Experience with the 
Specific M&S 
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5.4.9.1 Explanations 

The primary information to convey is the qualifications of the people developing, testing, and 
using this M&S. Depending on the size and complexity of the M&S activity, the developer, 
operator, and analyst roles may be accomplished by one person or by a team of people for each 
role. The qualifications example in table 14 can assist in organizing this infonnation. This table 
is only an example and may be tailored to the specific project. 

5.4.9.2 Examples 

Table 1 5, Example 1 : The M&S Developer, Operator, and Analyst are the Same Person, and 
table 16, Example 2: The M&S Developer, Operator, and Analyst are Different People, show 
examples for documenting People Qualifications. 


Table 15 — Example 1: The M&S Developer, Operator, and Analyst are the Same Person 


M&S Developer 

M&S Operator 

M&S Analyst 

CAS 

Factor 

Educational 

Background 

BS in Aerospace 
Engineering 

BS in Aerospace 
Engineering 

BS in Aerospace 
Engineering 

2 

Experience in M&S 
Discipline 

4 years 

4 years 

4 years 

Training for the 
Specific M&S 

Two 40-hour classes 
in MATLAB " 

Two 40-hour classes 
in MATLAB'* 

Two 40-hour classes 
in MATLAB* 

Experience with the 
Specific M&S 

2 years 

2 years 

2 years 

In the above example, the developer, operato 
single column. 

Table 16 — Example 2: The M&S Develo 

r, and analyst columns could be collapsed into a 

per, Operator, and Analyst are Different People 


M&S Developer 

M&S Operator 

M&S Analyst 

CAS 

Factor 

Educational 

Background 

BS in Computer 
Science 

BS in Aerospace 
Engineering 

MS in Mechanical 
Engineering 

3 

Experience in M&S 
Discipline 

5 years 

5 years 

1 0 years 

Training for the 
Specific M&S 

Five 40-hour classes 
in MATLAB* 

One 40-hour class in 
MATLAB* 

None 

Experience with the 
Specific M&S 

5 years 

5 years 

5 years 


5.4.9.3 Improving Credibility in People Qualifications 

Table 17, Achieving People Qualifications CAS Factor Levels, includes information from 
NASA-STD-7009 as to the aspects and evidence for achieving each credibility level for People 
Qualifications, along with suggested methods of how to attain each assessed level. The extensive 
work experience for Credibility Levels 3 and 4 is defined as follows: 
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a. Level 4: The individual or team lead has sufficient experience to mentor new and 
experienced practitioners on the specific technical and M&S disciplines without further technical 
oversight. 

b. Level 3: The individual or team lead has sufficient experience to mentor newcomers 
on the subject discipline (not necessarily the M&S) without technical oversight. 


Table 17— Achieving People Qualifications CAS Factor Levels 


Levels 


Evidence 

How Each Level is Achieved 

4 

Extensive 
experience in 
and use of 
recommended 
practices for this 
particular M&S 

Possesses an advanced 
engineering or science degree or 
extensive work experience in 
M&S, has extensive experience 
with the development and use of 
the M&S being reviewed, and has 
employed specific recommended 
practices relevant to current 
application 

Relevant to the specific technical M&S 
discipline: SME; extensive work 
experience in development and use; 
experience in the development and use 
of recommended practices 

3 

Advanced 
degree or 
extensive M&S 
experience and 
recommended 
practice 
knowledge 

Possesses an advanced 
engineering or science degree or 
extensive work experience, has 
general M&S training, has 
specific experience with the M&S 
being reviewed, and has been 
trained on specific recommended 
practices relevant to the current 
application 

Advanced degree 

Advanced M&S training 

Relevant to the specific technical M&S 
discipline: work experience in 
development and use; formal training 
and experience with the recommended 
practices 

2 

Formal M&S 
training and 
experience and 
recommended 
practice training 

Possesses an engineering or 
science degree, has received 
formal training in formulation of 
M&S and generic training in 
recommended practices for M&S, 
and has developed M&S products 

Formal technical education, training, 
and experience in the discipline 
relevant to the specific M&S and 
analysis 

Knowledge of discipline-specific 
recommended practices 

M&S development experience 

1 

Engineering or 
science degree 

i 

Possesses an engineering or 
science degree, has been 
introduced to the topic of M&S, 
and has been exposed to generic 
recommended practices in M&S 

Basic technical education, training, 
and experience are documented for the 
practitioners involved in the M&S and 
analysis. 

0 

Insufficient 

evidence 

Insufficient evidence 



5.4.10 Technical Review 

A Technical Review of an M&S is commonly referred to as a peer review, i.e., a thorough 
review of technical content by peers. In NASA-STD-7009, Technical Review is not a separate 
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factor in the overall M&S credibility assessment process but is a sub-factor for Verification, 
Validation, Input Pedigree, Results Uncertainty, and Results Robustness. 

Figure 47, Worksheet Item: Technical Review, indicates a single general question about the 
qualifications of the M&S practitioners. 



Item 

| ✓ | Result 

Comments 

M&S Credibility Assessment (Development - Usage (Analysis) ■ 

- Process | 


Technical Review 

Provide a summary of the Technical 
Reviews performed on this M&S/Analysis. 





Figure 47 — Worksheet Item: Technical Review 


Additional details to consider in the assessment of this sub-factor include the following: 

a. The list of the technical reviewers and their qualifications. 

b. The technical diversity in the review panel. 

c. The independence of the peer reviewers from the project. 

d. The formality of the Technical Review process, i.e., how it is planned, followed, and 
documented. 

e. The currency of the Technical Review with the current revision of the M&S. 

f. The comprehensiveness of the review for the M&S activity for all phases of the 
project, including independent technical activity, e.g., independent modeling, simulation, 
verification, validation, analysis. 

A summary of all Technical Review information should be included in the Results column, with 
details, explanatory notes, or pointers to other documentation annotated in the Comments 
column. 

5.4.10.1 Explanations , 

The level definitions of the Technical Review sub-factor focus on the degree of formality, 
technical expertise, and independence of the review undertaken. The formality and rigor of a 
review is characterized by the planning, process followed, and level of documentation. A formal 
Technical Review is well planned, followed, and documented. Reviewer qualifications and 
independence from the program/project should be considered. The independence of a review 
(internal to external) is characterized by how closely the panel members are to the 
program/project the M&S supports. In addition, the technical diversity of the reviewers is 
valuable in providing alternative perspectives to the problem domain. The comprehensiveness of 
the review entails how much of the M&S lifecycle was covered and to what level of detail. 
Finally, the completion date of each Technical Review should be documented. The date of the 
latest Technical Review, in conjunction with the change history of an M&S, indicates the 
currency of the review with respect to the current revision of the M&S. 
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This repetition across these CAS factors is beneficial in that numerous reviews are generally 
constituted throughout the program/project lifecycle, with each review focusing on particular 
topics. It would not be uncommon for different SMEs to be involved in the various reviews, 
depending upon their availability, areas of expertise, and focus of a particular review. 

Note: The Technical Reviews for M&S need not be constrained to the Development & 
Operations CAS factors (Verification, Validation, Input Pedigree, Results Uncertainty, and 
Results Robustness.) These five factors address the most technical aspects of an M&S; however, 
a complete M&S review could examine the additional factors under the Supporting Evidence 
heading, i.e.. Use History, M&S Management, and People Qualifications. 

5.4.10.2 Examples 

Examples will be developed and provided in a later revision. 

5.4.10.3 Improving Technical Review Credibility Assessment 

The method of assessing the quality and completeness of a Technical Review can vary, 
depending on the aspects of a project. While the assessment result may fall anywhere in that 
spectrum, the best and worst conditions may be anchored as shown in table 1 8, Aspects of 
Technical Review Assessments. 


Table 18 — Aspects of Technical Review Assessments 



No Evidence 


Best Possible 

Technical 

Expertise 

No relevant expertise 
documented 

<— > 

Reviewers are all educated, 
trained, and experienced with this 
type of M&S. 

Diversity in 
Expertise 

Technical diversity in 
review members not 
documented 


Expertise in Technical Review is 
a good mix within and outside the 
M&S domain. 

Level of 
Independence 

No independence from 
M&S project 

<— > 

Fully independent review panel, 
external to Agency with no ties to 
implementing organization 

Level of Formality 

No formal process 
documented 

<—> 

Planning, execution, and closure 
of review formally documented 

Phases of the M&S 
Lifecycle Covered 

No Technical Reviews 
documented 


All phases of M&S project 
reviewed 

Level of Detail 

Detail level of 
Technical Review not 
documented 


All details of M&S reviewed 

Currency with 
Revision of M&S 

M&S revision not 
documented for the 
review 


Review current to latest revision 
of M&S 
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The best possible Technical Review in all respects is rarely practical and depends on the 
criticality, budget, schedule, and reviewer availability. The purpose of the assessment of 
Technical Review is only to understand how well it was accomplished. 

Table 19, Achieving Technical Review CAS Factor Levels, shows the increasing gradation of 
Technical Review, with the highest level (4) reserved for reviews that include an independent 
evaluation of the individual factor. A formal review is documented and rigorous. An informal 
review is more ad hoc. NASA-STD-7009 assumes any external review is formal. 


Table 19 — Achieving Technical Review CAS Factor Levels 


Level 

Technical Review 

How to Achieve Each Level 

4 

Favorable external peer review accompanied 
by independent factor evaluation 

In general, better reviews are more 
independent, highly qualified with 
more technical diversity, more formal 
(planned and documented), encompass 
the full M&S lifecycle, cover more 
detail, current with the latest (or used) 
revision of the M&S. 

3 

Favorable external peer review 

2 

Favorable formal internal peer review 

1 

Favorable informal internal peer review 

0 

Insufficient evidence 

No technical review 


An unfavorable review at any one level does not allow one to achieve that particular level. For 
example, if an external peer review for Level 3 is unfavorable, a favorable formal internal peer 
review still achieves Level 2. 

5.5 Section 4 - NASA-STD-7009 Requirements & M&S Risk 

This last section includes two items for consideration of a complete NASA-STD-7009 based 
M&S assessment. First, it can be instructive to see how a given M&S activity has complied with 
each of the requirements in NASA-STD-7009, which relates to the requirements compliance 
matrix (Appendix C of NASA-STD-7009). Second, the M&S-based Analysis Risk item is a 
reminder of the two-fold role of risk in M&S-based analyses: the required use of 
NASA-STD-7009, based on the level of influence of the M&S on the consequences of the 
impending decision (in accordance with Appendix A of NASA-STD-7009) and an understanding 
and acceptance of the risk associated with the decision when the results from the M&S-based 
analysis are presented. The answer to the question “Is NASA-STD-7009 required?” is 
documented as an item in the Worksheet header. Tracking the program/project decision risks 
stemming from M&S-based analysis is captured in the M&S-based Analysis Risk item. 

5.5.1 NASA-STD-7009 Requirements Compliance 

The intent of NASA-STD-7009 is to establish routine and disciplined M&S processes. Section 4 
of NASA-STD-7009 enumerates the minimum set of requirements to that end, along with an 
additional set of recommendations in each sub-section. Understanding the requirements that are 
and are not satisfied offers additional supporting evidence to the amount of rigor associated with 
the development of the M&S and its use in analysis. This associated Worksheet item establishes 

APPROVED FOR PUBLIC RELEASE — DISTRIBUTION IS UNLIMITED 


109 of 130 














N AS A-HDBK-7009— MONTH DD, 2013 


the importance of recording the level of compliance with these requirements and 
recommendations. 

An understanding of the extent of compliance with the requirements of NASA-STD-7009 is 
needed. This is intended to be a high-level perspective supported by the Compliance Matrix in 
Appendix C of NASA-STD-7009. Consideration should be given to whether and how well the 
requirements were satisfied. Waivers to the requirements in NASA-STD-7009 should be 
documented. 

While it is desirable that all requirements and recommendations are satisfied, program/project 
constraints can limit full compliance. Therefore, a view of adherence is not the best way to report 
this compliance. It is more important for an M&S activity to provide information directly related 
to the analysis than to have satisfied all of the check-the-box requirements. The Results column 
can be used to record an indication of compliance, e.g., percent completed, and then the 
Comments column can be used to note significant exceptions or accomplishments (figure 48, 
Worksheet Item: Requirements Compliance). 



Item 

| V | Result 

| Comments 

NASA-STD-7009 Requirements & M&S Risk 

Requirements 

Compliance 

Give details on non-compliances with 
NASA-STD-7009 and their consequences. 





Figure 48 — Worksheet Item: Requirements Compliance 


5.5.1. 1 Explanations 

Appendix C of NASA-STD-7009 provides a template for assessing requirements compliance. 
The first column lists the requirements; the second column is for recording the compliance status 
(C-compliant, NC-not compliant, N/A-not applicable); and the third and fourth columns can be 
used to record the method of verifying compliance and the evidence of compliance, respectively. 

In practice, such a discrete view of compliance may not be the best approach. Critical decisions 
sometimes are made independent of the schedule governing the development of models, 
execution of simulations, final analysis, and documentation. Accordingly, degrees of partial 
compliance could be considered. For example, the technical work with respect to a particular 
CAS factor (and its associated group of requirements) may be accomplished and documented in 
presentation format, while the formal documentation necessary to satisfy all the requirements is 
still in progress. Also, the uncertainty and sensitivity analyses may involve a large number of 
parameters, only some of which had been evaluated at that point. In cases like these, it may be 
acceptable to make a judgment call and choose either “Compliant” or “Not Compliant” or use 
modifiers, e.g., essentially or partially, as long as they are defined. The other columns can then 
be used to record the status of the M&S effort relative to achieving full compliance. 

5.5. 1.2 Examples 

Table 20, NASA-STD-7009 Compliance Matrix Partial Sample, shows example entries in the 
Compliance Matrix from Appendix C of NASA-STD-7009. 
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Table 20 — NASA-STD-7009 Compliance Matrix Partial Sample 


Requirements 

Compliance 

Status 

(C, NC, N/A) 

Method 

Evidence 

Requirement 4.1 .1 - Shall document the risk 
assessment for any M&S used in critical 
decisions 

NC 

N/A 

Risk Assessment 
not accomplished 

Requirement 4.1.2- Shall identify and 
document those M&S that are in scope 

C 

Set of M&S used 
and architecture 
documented in 
the XYZ M&S 
Plan 

XYZ M&S Plan - 
Appendix A: 
Analysis 
Architecture 


NASA-STD-7009 also includes a number of recommendations that are not listed in the 
Compliance Matrix. Using the format from Appendix D of this Handbook, compliance with 
these recommendations and other information could be captured. 

5.5.2 M&S-based Analysis Risk 

One of the basic challenges faced when presented with any analysis from which a decision is 
required is to understand the risks involved. These risks culminate in the end-state resulting from 
the decision and the subsequent actions or operations proceeding from that decision. These risks 
may be rooted in the modeling, simulation, and/or analyses producing the results on which to 
base a decision. The presentation of the M&S-based analysis risks for a decision with respect to 
the consequences of the impending decision is necessary. This Worksheet item addresses the 
risks of basing the decision on the M&S-based analysis and use of the program’s/project’s risk 
assessment and decision-making process to track specific M&S-based analysis risks. 

Note: The intent of this Worksheet item is different from the M&S Risk Assessment defined in 
Appendix A of NASA-STD-7009, which determines what M&S are required to use 
NASA-STD-7009. That requirement is based on the influence the M&S will have in the decision 
process and the severity of the decision consequence and is annotated in the header portion of the 
Worksheet. 

The essence of this item is thus captured in the question included in this Worksheet item (figure 
49, Worksheet Item: M&S-based Analysis Risk): “What are the risks of basing this decision on 
the M&S-based analysis?” 



Item 

| ✓ | Result 

| Comments 

NASA-STD-7009 Requirements & M&S Risk 

M&S-based Analysis 

What are the risks of basing this decision on 





Risk 

the M&S-based analysis? 






Figure 49 — Worksheet Item: M&S-based Analysis Risk 


Program/project risks are often quantified through an M&S process. An understanding of how 
the M&S-based analysis quantifies program/project risk and of what risks are inherent in the 
M&S techniques is needed to ensure a complete understanding of a given decision. Two key 
ideas to keep in mind are: 
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• Application of M&S practices to quantify program/project documented risks. 

• Tracking of risks inherent in the M&S implementation. 

Potential entries in the Worksheet for this item are: 

• Whether the risks associated with the results are presented. 

• Whether the risks are acceptable for this project. 

5.5.2. 1 Explanations 

A program/project may accept various levels of risk, depending on the mission type and 
criticality. NPR 8000.4, Agency Risk Management Procedural Requirements, evolves NASA’s 
risk management to entail two complementary processes: Risk-Informed Decision Making 
(RIDM) and Continuous Risk Management (CRM). RIDM is intended to inform systems 
engineering decisions through better use of risk and uncertainty information in selecting 
alternatives and establishing baseline performance requirements. CRM is used to manage risks 
over the course of the development and implementation phases of the lifecycle to assure that 
requirements related to safety, technical, cost, and schedule are met. NASA provides guidelines 
for the RIDM process in NASA SP-201 0-576. The guidelines for the entirety of the NASA risk 
management approach, including both RIDM and CRM, are provided in NASA/SP-201 1-3422, 
NASA Risk Management Handbook. In programs/projects where a Risk Management Plan 
(RMP) is in place, the risk addressed by an M&S process is most likely posed in the RMP 
implementation. M&S is one analysis process by which RIDM-/CRM-enumerated performance 
measures of a risk may be probabilistically quantified. M&S activities can provide analysis and 
justification for risk acceptance and mitigating actions against program/project risks. This 
process assumes adherence to sound M&S practices, e.g., V&V and M&S integrity by CM, and 
the appropriate M&S applicability with regard to quantifying a risk’s performance measure. The 
RIDM/CRM process helps assure the M&S results are correctly integrated into the 
program/project decision-making process. 

5.5.2.2 Examples 

5.5.2.2.1 Example 1 : Application of M&S practices to quantify program/project documented 
risks 

An example of a tracked program/project risk could be: Ice formed on the Space Shuttle’s 
External Tank on the launch pad can dislodge and impact the Orbiter during ascent. There is a 
probability the resulting structural damage could result in the loss of the Orbiter and crew. 

Questions to ask during a typical review meeting are: 

• How was M&S used to assess program/project risk performance measures? 

• What are the risks associated with this decision? 
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• How were the performance measures for this risk integrated into the program/project 
decision-making process? 

• Has the M&S-based analysis adequately addressed the risks associated with the decision? 

To answer these questions, the risks and associated performance measures should be assessed, 
possibly by using the M&S and associated analyses. In the previous example, the ice impact on 
an Orbiter’s leading edge may be one of several risks quantified by an M&S approach that needs 
to be discussed in the program/project reviews. Answering these questions gives an indication 
about how comprehensive the M&S implementation and analysis were with regard to 
program/project needs. The results of the assessment should then be traceable to the risk- 
informed decision. 

Insight into the communication of the risk of using the results of an M&S-based analysis can be 
provided by the CRM process’s risk management matrix. NASA/SP-2007-6105 discusses CRM 
and RIDM in context with use of a typical risk management matrix (figure 50, Example Risk 
Matrix). This type of matrix is often used to assess a risk’s ranking in the program/project risk 
posture. 



1 2 3 4 5 

Consequences 

Figure 50 — Example Risk Matrix 


5.5.2.2.2 Example 2: Tracking of risks inherent in the M&S 

If the M&S process is seen to have its own risks, a statement defining those risks should be 
documented in the program/project risk management process. 

As an example, lack of validation data for the tolerances in the Shuttle Orbiter wing leading edge 
damage model could lead to inadequate predictions in the severity of damage from the full range 
of known input boundary conditions. 
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This prompts review questions such as: 

• Were the risks inherent to an M&S implementation added to the program/project risk 
management process? 

• What modifications to the M&S validation process could be performed to increase 
confidence of the M&S assessment? 

• What was the outcome of the risk mitigation process? 


APPROVED FOR PUBLIC RELEASE — DISTRIBUTION IS UNLIMITED 


1 14 of 130 



NASA-HDBK-7009— MONTH DD, 2013 


6. SUMMARY 

While NASA-STD-7009 and this Handbook are provided to encourage good practices in M&S, 
not all M&S activities have to adhere to them. The required use of NASA-STD-7009 is 
determined by a risk assessment in accordance with Appendix A of NASA-STD-7009. The 
completeness or degree of compliance and expectations in credibility assessment are chosen by 
program/project management in collaboration with the Technical Authority in accordance with 
current NASA governance. 

The Worksheet introduced in this Handbook is an aid to implementing a rigorous M&S process 
and a more complete and standardized reporting of M&S-based analyses. This reporting 
includes: 

a. Background and contextual information about the RWS. 

b. The basis for the use of the M&S for the analysis. 

c. The reporting of results with a statement of uncertainty and caveats to the analysis. 

d. An understanding of results credibility as defined by NASA-STD-7009. 

e. The level of compliance with the requirements of NASA-STD-7009. 

f. An assessment of the risk associated with basing a decision on the analysis. 

This information is intended to guide a more complete discussion of an M&S-based analysis, 
though is not expected to cover every aspect of all types of M&S. Each type of M&S, 
engineering discipline, system, and project has a uniqueness that may not be included. 
Discipline-specific M&S guides and program/project management requirements and practices 
should also be consulted. 
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APPENDIX A 
THE WORKSHEET 

A.l Purpose and/or Scope 

This appendix provides guidance in the form of a full-page version of the M&S Assessment 
Worksheet (figure 51, NASA-STD-7009 M&S Assessment Worksheet), as discussed in section 5 
of this Handbook. A spreadsheet version of the M&S Assessment Worksheet can be downloaded 
from https://standards.nasa.gov/released/nasa/NASA-HDBK-7009 Worksheet.xlsx . 


APPROVED FOR PUBLIC RELEASE — DISTRIBUTION IS UNLIMITED 


1 16 of 130 


NASA-HDBK-7009— MONTH DD, 2013 


A.2 NASA-STD-7009 M&S Assessment Worksheet 



NASA-STD-7009 
M&S Assessment Worksheet 

— — 

System: 

M&S: 

! 

2. 

1 



In Development / Operational 

Sub-System, Dement, or Aspect of System Under Analysis: 

M&S Responsible Party: 

System life cycle Phase: 

Subject of Analysis (e g. Production. Crowd Ops, FSgH, Mnsbn. Entry. Descent, Lands*): 

Responsibility Chaim P/P Mgt & Tech Authority 

Date: 

Risk Assessment Result (per NASA-STD-7M9 Appendix Aft 

7009 Use Is Requrcd / 7009 Uae Is NOT Reqiied / Not Performs* 

Item | ✓ 

Result 

Comments 

System & Analysis Frameworks 

Real World 
System /Problem 

What is the real world system? 




What is its environment? 




What is the probfem/deemod? 




Model / A bs tract ions / 
Assumptions 

What is the M&S approach? 




What's included in the M&S, nefodng 
model environment influences? 




Is there anything significant to this analysis 
not included in the M&S or scenario*? 




nchafed si the MAS and Amlysn? 




System - Model 
Match 

How wel docs this M&S represent the 
Real Worid System /Problem at hand? 




How wel does dm M&S produce the 
results necessity for this mlysB? 




M&S-bssed Analysis Results & Caveats 

Ultimate 

byicaaafrwfe? 




How wel do the analysis results address the 




Uncertainty 
in Estimate 

Whatare the magnitudes of the incettainties 
in the results of this analysis? 




Caveats 

What are the caveats to the analysis with 
tha M&S? 




M&S Credibility Assessment (Development • Usage (Analysis) 

• Processl 

Overall Credibility 

What are the ovcral results of the M&S 
Credbity Assessment? 




Verification 

How (wel) docs the M&S implementation 
match the conceptual specification? 




Validation 

How wel did test predictions using the 
M&S match referent data? 




How close is the referert to the real- worid 
system, including its envronment? 




Input Pedigree 

What is the qualty of the Input Data? 




How authoritative is the Input Data for this 
analyse? 




Results Uncertainty 

What methods are teed to analyse the 
unccrtarty in the results of this analysis 
(inckjdng soirees and propagation)? 




Results Robustness 

What are the significant sensitivities of the 
M&Srewb? 




How thoroughly are the sensitivities known? 




Use History 

How have the current M&S been 
previously used? 




MAS Management 

What formal processes were used in the 
development & use of dm M&S? 




People Qualifications 

What are the q unifications & experience of 
the people devebping. testing. & using this 
M&S? 




Technical Review 

Provide a summary of the Technical 
Reviews performed on this MAS/Analysk 




NASA-STD-7009 Requirements & M&S Risk 

Rapiirements 

Compliance 

N ASA-STD-7009 and their consequences 




M AS-based Analysis 
Risk 

What are the risks ofbasiqg dm decaiou on 
the M&S- based amlyss? 





Figure 51 — NASA-STD-7009 M&S Assessment Worksheet 
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APPENDIX B 
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APPENDIX C 

RELATIONSHIP OF NASA-STD-7009 TO NPR 7150.2 
C.l Purpose and/or Scope 

Models are usually implemented in software. This appendix provides guidance in the form of a 
discussion of the relationship between the requirements and guidance for NASA-STD-7009 and 
the requirements and implementation plans for NPR 7150.2. 

C.2 Embedded Models and Simulations 

The relevant guidance is from section 1.2 of NASA-STD-7009: “This standard does not apply to 
M&S that are embedded in control software, emulation software, and stimulation environments. 
However, Center implementation plans for NPR 71 50.2 should specifically cover embedded 
M&S, and address such M&S-specific issues as numerical accuracy, uncertainty analysis, 
sensitivity analysis, M&S verification, and M&S validation.” 

NASA-STD-7009 is mentioned in section 3.4.6 of NPR 7150.2, which reads, “The project shall 
verify, validate, and accredit software models, simulations, and analysis tools required to 
perform qualification of flight software or flight equipment. [SWE-070].” In NPR 7150.2, a 
software engineering (SWE) number designates a requirement. 

Note that Center processes address issues such as verification, validation, numerical accuracy, 
uncertainty analysis, and sensitivity analysis for software implementations of M&S. Information 
regarding V&V techniques and the analysis of M&S can be found in NASA-STD-7009. 

This note is meant to refer to all germane requirements in NPR 7150.2, not just to V&V of M&S 
for qualification of flight software or flight equipment. Accordingly, the implementation plans 
for NPR 7150.2 should address the verification, validation, numerical accuracy, uncertainty 
analysis, and sensitivity analysis of embedded M&S. 

The relevant requirements from NASA-STD-7009 are 4.2.6 and 4.4.1 to 4.4.9. Each of these 
requirements begins with the phrase “Shall document.” None of them requires that any specific 
activity be performed other than the relevant documentation. As NASA-STD-7009 states in 
section 4.1 : “...whatever was done is to be documented, and if nothing was done a clear 
statement to that effect is to be documented. ” 

Since Center implementation plans for NPR 7150.2 vary in format and content, a specific 
prescription for dealing with embedded M&S in such a plan cannot be given; rather, the 
recommendation is for each implementation plan to mention the special documentation required 
for embedded M&S, in accordance with requirements 4.2.6 and 4.4. 1 to 4.4.9 of 
NASA-STD-7009. 
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C.3 Other Models and Simulations 

/ 

For all other M&S deemed by the risk assessment to be in scope of NASA-STD-7009, the 
requirements of both documents must be satisfied. From the perspective of NASA-STD-7009, 
some requirements in NPR 7150.2 are not applicable to M&S, while some are supplemental to or 
subsets of the requirements in NASA-STD-7009. 

Table 21, Relationship of NPR 7150.2 to NASA-STD-7009, indicates the specific relationship of 
each requirement in NPR 7150.2 to NASA-STD-7009. The following key is to the remarks in the 
fourth column of table 21 : 

a. NA: The NPR requirement is not germane to M&S per se. It may, for example, be a 
requirement on the NASA Centers. (See SWE-005 of NPR 7150.2: “Each Center shall establish, 
document, execute, and maintain software processes.”) 

b. Supplemental: The NPR requirement is relevant to software for M&S but exceeds 
any requirement in NASA-STD-7009. For example, “The project shall implement, maintain, and 
execute the software plan(s)” (SWE-014 of NPR 7150.2) has no counterpart in 
NASA-STD-7009, which just requires a plan (albeit for the M&S as a whole and not just the 
software) (Requirement 4. 1 .4) but has no requirement for this plan be implemented, maintained, 
or executed. 

c. Subset of Requirement X: The NPR 7 1 50.2 requirement is part of a requirement in 
NASA-STD-7009: Requirement 4. 1 .4 requires a plan for the M&S that includes such software 
aspects as verification and configuration management but has additional requirements on M&S- 
specific aspects with no counterpart in NPR 7150.2. 


Table 21 — Relationship o 

NPR 7150.2 to NASA-STD-7009 

Section of NPR 
7150.2 

Requirement Descriptor 

SWE# 

NASA-STD-7009 Relationship 

Preface 

Effective date 

1 

Scope statement implicitly includes M&S 
implemented in software. 


Agency software initiative 

2 

NA 


Center plan 

3 

NA 


Benchmark 

4 

NA 

Organizational 

Capability 

Software processes 

5 

The Note in 3.4.6 of NPR 7150.2 means the 
topics in the Note are to be addressed in Center 
processes. 


List of Agency's programs 
and projects containing 
software 

6 

NA 


Software plans 

13 

Subset of Requirement 4.1.4 



14 

Supplemental 

SW Lifecycle 
Planning 

Cost estimation 

15 

Supplemental 

Schedule 

16 

Supplemental 

Training 

17 

Supplemental to Requirement 4.6.2(a) 


Reviews 

18 

Supplemental 


Software development 

19 

Supplemental 
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Section of NPR 
7150.2 

Requirement Descriptor 

SWE# 

NASA-STD-7009 Relationship 


lifecycle or model 



Software classification 

20 

Supplemental 


Software classification 
changes 

21 

Supplemental 

Software assurance 

22 

Supplemental 

Software safety 

23 

Supplemental 

Plan tracking 

24 

Supplemental 

Corrective action 

25 

Supplemental 

Changes 

26 

Supplemental 

OTS SW 

COTS, GOTS, MOTS 

27 

Supplemental 

v&v 


28 

Subset of Requirement 4.1.3(e) 


29 

Subset of Requirement 4. 1 .3(e) 

Verification results 

30 

Subset of Requirements 4.4.1 - 4.4.3 

Validation results 

31 

Subset of Requirements 4.4.4 - 4.4.6 

Project 

Formulation 

CMMC™ levels for classes 
A, B, and C software 

32 

Supplemental 

Acquisition assessment 

33 

Supplemental 

Acceptance criteria 

34 

Subset of Requirement 4. 1 .3(a) 

Supplier selection 

35 

Subset of Requirement 4.1.4 

Software processes 

36 

Supplemental 

Milestone 

37 

Supplemental 

Acquisition planning 

38 

Supplemental 

Government Insight 

Insight into software 
activities 

39 

Supplemental 


40 

Supplemental 

Open source notification 

41 

Supplemental 

Electronic access to source 
code 

42 

Supplemental 

Supplier 

Monitoring 

Track change request 

43 

Supplemental 

Software measurement data 

44 

Supplemental 

Joint audits 

45 

Supplemental 

Software schedule 

46 

Supplemental 

Traceability data 

47 

Supplemental 

Solicitation 

48 

Supplemental 

Software 

Requirements 

Development 

Document 

49 

Subset of Requirement 4.1.3 

Software requirements 

50 

Supplemental 

Flow-down and derived 
requirements 

51 

Supplemental 

Bi-directional traceability 

52 

Supplemental 

Software 

Requirements 

Management 

Manage requirements 
change 

53 

Supplemental 

Corrective action 

54 

Supplemental 

Requirements validation 

55 

Supplemental 

Software Design 

Document design 

56 

Supplemental 

Software architecture 

57 

Supplemental 

Detailed design 

58 

Supplemental 

Bi-directional traceability 

59 

Supplemental 

Software 

Implementation 

Design into code 

60 

Supplemental 

Coding standards 

61 

Supplemental 

Unit test 

62 

Supplemental 
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Section of NPR 
7150.2 

Requirement Descriptor 

SWE# 

NASA-STD-7009 Relationship 


Version description 

63 

Subset of Requirement 4.2.9 

Bi-directional traceability 

64 

Supplemental 

Software Testing 

Plan, procedures, reports 

65 

Supplemental 

Perform testing 

66 

Supplemental 

Verify implementation 

67 

Supplemental 

Evaluate test results 

68 

Subset of Requirements 4.4.1 - 4.4.6 

Document and track 
defects 

69 

Supplemental 

Models, simulations, tools 

70 

Supplemental 

Update plans and 
procedures 

71 

Supplemental 

Bi-directional traceability 

72 

Supplemental 

Platform or hi-fidelity 
simulations 

73 

NA to M&S in scope 

Software 
Operations, 
Maintenance, and 
Retirement 

Document maintenance 
plans 

74 

Subset of Requirement 4. 1 .4 

Plan operations, 
maintenance, and 
retirement 

75 

Subset of Requirement 4.1.4 

Implement plans 

76 

Supplemental 

Deliver software products 

77 

Supplemental 


As-built documentation 

78 

Supplemental 


Develop configuration 
management plan 

79 

Supplemental 

Software 

Configuration 

Management 

Track and evaluate changes 

80 

Supplemental 

Identify software 
configuration items 

81 

Supplemental 

Authorizing changes 

82 

Supplemental 

Maintain records 

83 

Supplemental 

Configuration audits 

84 

Supplemental 

Implement procedures 

85 

Supplemental 

Risk Management 

Continuous risk 
management 

86 

Supplemental 

Peer Reviews/ 
Inspections 

Requirements, test plans, 
design, and code 

87 

Supplemental 


88 

Supplemental 

Basic measurements 

89 

Supplemental 

Software 

Measurement 

Objectives 

90 

Supplemental 

Software measurement 
areas 

91 

Supplemental 

Collection and storage 

92 

Supplemental 

Analyze data 

93 

Supplemental 

Report analysis 

94 

Supplemental 

Software measurement 
system 

95 

NA 

Objectives and procedures 

96 

NA 

Best Practices 


98 

NA 


99 

NA 

Training 


100 

NA 
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Section of NPR 
7150.2 

Requirement Descriptor 

SWE# 

NASA-STD-7009 Relationship 


training 



Software training plan 

101 

NA 

Software 

Documentation 

Requirements 

Software development/ 
management plan 

102 

Supplemental 

Software configuration 
management plan 

103 

Supplemental 

Software test plan 

104 

Supplemental 

Software maintenance plan 

105 

Supplemental 

Software assurance plan 

106 

Supplemental 

Center software training 
plan 

107 

NA 

Center software 
engineering improvement 
plan 

108 

NA 

Software requirements 
specification 

109 

Supplemental 

Software data dictionary 

110 

NA to M&S in scope 

Software design 
description 

111 

Supplemental 

Interface design description 

112 

NA to M&S in scope 

Software change 
request/problem report 

113 

Supplemental 

Software test procedures 

114 

Supplemental 

Software users manual 

115 

Supplemental 

Software version 
description 

116 

Supplemental 

Software metrics report 

117 

Supplemental 

Software test report 

118 

Supplemental 

Software inspection/ 
peer review/inspections 

119 

Supplemental 

Tailoring of 
Requirements 

Submit generic waiver 
request 

120 

Supplemental 

Document approved 
alternate requirements 

121 

Supplemental 

Designation of 
Engineering 
Technical Authority 

Center-level Engineering 
Technical Authority 
approval 

122 

NA 

Compliance 

Direction for Technical 
Authority 

124 

NA 

Compliance matrix 

125 

Supplemental 

Considerations for waivers 

126 

NA 

Review of "P (Center)" 

127 

NA 

Compliance records 

128 

NA 

Check compliance 

129 

NA 

Software Lifecycle 
Planning 

Software safety plan 

130 

Supplemental 

IV&V Plan 

131 

Supplemental 

Classification assessment 

132 

NA 

Software safety 
determination 

133 

Supplemental 

Safety-critical software 
requirements 

134 

Supplemental 

Software 

Static analysis 

135 

Supplemental 
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Section of NPR 
7150.2 

Requirement Descriptor 

SWE# 

NASA-STD-7009 Relationship 

Implementation 

Validation of software 
development tools 

136 

Supplemental 

Software Peer 
Reviews/ 
Inspections 

Peer review/ inspections of 
software plans 

137 

Supplemental 

Software 

Documentation 

Requirements 

Software safety plan 
contents 

138 

Supplemental 

Compliance 

“Shall”: statements in NPR 
7150.2 

139 

Supplemental 

“P (Center)” 

140 

Supplemental 
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APPENDIX D 

RECOMMENDATIONS COMPLIANCE MATRIX 
(NOT INCLUDED IN NASA-STD-7009) 

D.l Purpose and/or Scope 

This appendix provides guidance in the form of table 22, Recommendations Compliance Matrix. 

L . . 

D.2 Recommendations Compliance Matrix 

Recommendations from NASA-STD-7009 include the following: 


Table 22 — Recommendations Compli 

lance Matrix 

Recommendations 

Compliance 

Status 

(C, NC, N/A) 

Method 

Evidence 

Rec. 4.3a: The relevant characteristics of the 
system that is modeled should be documented. 




Rec. 4.3b: CM records should contain test cases 
that span the limits of operation for the M&S 
defined by the program or project. Test cases are 
defined as benchmark I/O sets used to verify 
proper execution of the M&S. 




Rec. 4.3c: The simulation should fail in a 
manner that prevents misuse and misleading 
results. 

1 . The simulation should provide messages 
that detail the failure mode and point of 
failure. 

2. The analyst should document and 
explain all failure modes, points of 
failure, and messages indicating such 
failures. 




Rec. 4.4: The responsible party should 
document: 




a: Any aspects of M&S that have not been 
verified. 




b: Any aspects of M&S that have not been 
validated. 




c: If any significant physical processes, 
effects, scenarios, or environments have not 
been considered in the uncertainty quantification 
analysis. 




Rec. 4.5: Recommended Practices should be 
identified for: 




a: Input data and V&V 
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Recommendations 

Compliance 

Status 

(C, NC, N/A) 

Method 

b: A quantified method of tracking adherence 
to Recommended Practices 




c: The purposes and objectives for the M&S 
and their pedigrees 


d: V&V processes for the M&S 


e: Uncertainty quantification methods for the 
M&S 


f: Understanding the disciplines incorporated 

in the M&S 

g: Analyzing and interpreting the M&S 
results, including documentation of inference 

guidelines and statistical processes used 

h: Recognizing and capturing the need for any 

changes or improvements in the M&S 

i: Reporting procedures for results 

j: Identifying best practices for user interface 
design to constrain the operation of the 
simulation to within its limits of operations 
Rec. 4.6: Recommended training topics for 
developers, operators, and analysts of M&S 
include: 


a: The intended use of limits of operation for 

models 

b: CM requirements 

c: Documentation requirements as specified in 
sections 4,2, 4,3, and 4,4 of NASA-STD-7009 
d: How to recognize unrealistic results from 
simulations 


e: Feedback processes to improve M&S 
processes and results, including providing 
feedback for results that are not credible, are 

unrealistic, or defy explanation 

f: Sensitivity analysis 


Uncertainty quantification 


h: V&V 


i: How to report simulation results to decision 

makers 

j: Statistics and probability 

k: Discipline-specific recommended practices, 
other applicable Agency policy, procedural 

requirements, and Standards 

1: Basic structures, mathematics, assumptions, 

and abstractions 

Rec. 4.7: Obtain additional insight into the 
credibility of M&S results by applying the 
process in Appendix B.5 of NASA-STD-7009 to 
calculate and report any gaps between the 
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Recommendations 

Compliance 

Status 

(C, NC, N/A) 

Method 

Evidence 

achieved scores and the program-/project- 
defmed threshold scores for each of the factors. 




Rec. 4.8: Reports to decision makers should: 




a: Include concluding remarks stating whether 
the M&S results are credible enough for the 
intended use 




b: Identify how to access more detailed 
backup material, including high-level 
descriptions of the models used and key 
assumptions for limits of validity 



• 

c: Be placed in the CM system 




d: Summarize deviations from established 
recommended practices 




e: Include dissenting technical opinions 
regarding the credibility of the results or any 
recommended actions 




f: Convey serious concerns of the developers 
and analysts about M&S to project managers 
(and decision makers, if appropriate) as soon as 
they are known 
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APPENDIX E 

TECHNICAL WORKING GROUP 
E.l Purpose and/or Scope 

This appendix provides guidance in a list of the individuals in the Technical Working Group who 
drafted this Handbook. 


E.2 Technical Working Group 


Name 

Martin Steele 

(Office of Primary Responsibility Designee) 

Tim Barth 

Mike Carney 
Jeff Cerro 
Howard Conyers 
Steve Comford 
Zack Crues 
Tim Crumbley 
Ken Johnson 

Mary Livingston 
Barbara McKissock 
Gary Mosier 
Keith Niehuss 
Bill Othon 
Tom Zang 


Center 

Kennedy Space Center 

NASA Engineering and Safety Center 
(Kennedy Space Center) 

Kennedy Space Center 

Langley Research Center 
Stennis Space Center 
Jet Propulsion Laboratory 
Johnson Space Center 
Marshall Space Flight Center 

NASA Engineering and Safety Center 
(Marshall Space Flight Center) 

Ames Research Center 
Glenn Research Center 
Goddard Space Flight Center 
Marshall Space Flight Center 
Johnson Space Center 
Langley Research Center 


For further questions or guidance in the use of this Standard, contact the Office of Primary 
Responsibility or other Center representatives listed above. 
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